Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago.
(The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)
In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.
I'll add more as we find out more, but it probably won't be till later this afternoon PST.
shlomo_z 16 hours ago [-]
Crazy that Dang literally manages HN in his sleep!
We all knew that but I haven't seen any confirmation before this.
easterncalculus 10 hours ago [-]
I like hacker news but I don't think this site is worth getting paged over lol
archon810 5 hours ago [-]
You might be underestimating HN's popularity.
locknitpicker 4 hours ago [-]
> You might be underestimating HN's popularity.
I think you're confusing popularity with criticality. I'm sure everyone in here can withstand a few hours without browsing the page.
bayindirh 2 hours ago [-]
If you like the thing you're managing, then its health is critical for you, not your users.
It's dang's baby at this point, and this is a good thing, as long as HN doesn't affect his life in ways he doesn't want.
locknitpicker 2 hours ago [-]
> If you like the thing you're managing, then its health is critical for you, not your users.
Get a grip and go touch some grass. Even FANGs understand the concept of business hours SEV.
bayindirh 8 minutes ago [-]
I have a pretty firm grip on life and touch plenty of grass both literally and figuratively.
However, when something I care about crashes and burns once in a blue moon, I make sure to put the fire out, at least to make it survive till regular hours. Things I care about can be both business and personal, and nobody bugs me for them.
Maybe we shouldn't make any assumptions about people we don't personally know, while we are at it.
dang 16 hours ago [-]
failing to manage HN in my sleep is more like it
jonny_eh 15 hours ago [-]
Your sleep is more important than our work distraction.
seemaze 10 hours ago [-]
I was curiously productive this morning..
koakuma-chan 9 hours ago [-]
I took a shower for the first time in one week
bigbuppo 3 hours ago [-]
Bet that felt great.
fouc 6 hours ago [-]
looks like a redditor snuck into HN
arnavpraneet 5 hours ago [-]
came back just in time for me to spend the first hour of my work
zenoprax 14 hours ago [-]
Which is fine! I don't mind if it's down for a few hours. It reminds me that it's just a place to stop by for a bit before moving on. Like a digital coffee shop that sometimes has a leaky pipe and isn't open right at 7am.
I hope it doesn't change (much).
wkat4242 10 hours ago [-]
You're still a miracle worker. Single-handedly managing a well-known fully user-contributed site not just technically but moderation in contentious times like these and still keeping it working well and encouraging a positive user community can't be an easy task.
No worries, please take care of your sleep and thanks for all your hard work
dijit 16 hours ago [-]
We all have our moments, and I personally consider HN to be “best effort”, almost like a volunteer project. I’m not certain I’m correct: but thats the optics I have so my expectations are adjusted to that.
So don’t beat yourself up please.
When I worked for “SaaS unicorn” we typically had multiple levels of escalation, and acknowledging would have done nothing because the alarm would continue firing until fixed. Not sure what’s changed in 15 years of ops, I had assumed it would be better now- I can’t imagine silencing an alert totally by acknowledging it- if its still occurring.
I’m totally fine with how you handled it, if anything I am thankful. But that seems to be a system I would improve if I had the time.
“mute” is different than “resolve” to me, and both should exist. (Where mute is an acknowledgement of an issue as ongoing.)
giancarlostoro 15 hours ago [-]
Yeah we don't exactly pay to be on HN, not much to complain about. I appreciate everyone who works on HN.
jacquesm 10 hours ago [-]
We pay with content and with the fact that we attract the talent that eventually ends up powering ycombinator investment rounds.
phantasmish 9 hours ago [-]
It’s ad-supported. Any post with comments disabled is definitely an ad. Probably a lot of the others are, too.
abustamam 5 hours ago [-]
Your comment makes me realize that I consume HN differently than many others, because I've never seen a post with comments disabled and I've been around here for at least ten years. It's not that I don't think they don't exist — they obviously do because you're mentioning them. I've just never encountered one, primarily because I don't casually browse HN, ever. I subscribe to a pushbullet channel that notifies me when a post hits 500 up votes. That's it. The list of submissions on the home page (even on reddit) is just overwhelming to me so I use the pushbullet channel as a sort of community curated "best of" or "trending" signal.
Not to say that I don't procrastinate or waste time doing other nonsense. I can definitely spend a lot of time reading HN comments, as I'm doing right now.
Anyway,anyone who finds themselves with a problem with HN should try that out :)
mysterydip 7 hours ago [-]
I assumed the main purpose was to show off the ycombinator batches when they launch.
scottlamb 15 hours ago [-]
This. If it were a business-critical money fountain, I'd expect follow-the-sun SRE coverage. I don't think it is, so I can probably accept drinking my morning coffee without scrolling HN once in a while. There's only so much one can beat oneself up about a slow/incorrect response when the on-call is handled by what, just one person? maybe two people in the same time zone?
(Might be wise though to have PagerDuty configured to re-alert if the outage persists.)
notachatbot123 16 hours ago [-]
And that is a good thing. Sleep tight!
sizzle 7 hours ago [-]
Time to train an AI agent on your moderation activity and get some well deserved sleep!
dang 5 hours ago [-]
We're working on it! well, some of it.
I'm pretty happy with how it's developing—the trendline is promising—but not ready to rely on it in prod yet.
nubinetwork 10 hours ago [-]
I was starting to think you never slept, I remember that one time I emailed you at 1am. :)
commandersaki 16 hours ago [-]
Do you have nightmares of failing to manage HN when you sleep too?
djmips 11 hours ago [-]
I appreciate what you do. Hope you got some rest when it was all over.
feczeri_c 10 hours ago [-]
You deserve a lot of rest!
sailfast 15 hours ago [-]
Yeah, I mean how dare you?! I pay good money for high uptime SLAs! :)
qingcharles 14 hours ago [-]
I was today years old when I found out Dan sleeps.
gaudystead 13 hours ago [-]
I was today years old when I found out that dang's first name is Dan
anonymous908213 12 hours ago [-]
You'll never guess what letter dang's last name starts with.
smolder 12 hours ago [-]
A as in Ang, clearly.
qingcharles 12 hours ago [-]
No, he's Asian. The n is doing double-duty. His last name is Ng :p
fragmede 8 hours ago [-]
That's exactly what a French person with the last name of Les would say!
sizzle 7 hours ago [-]
What if Dang made an AI agent of himself for when he sleeps?
utopcell 8 hours ago [-]
By demonstration he didn't.
rldjbpin 34 minutes ago [-]
> The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users.
it is a shame that it needs to be this way. as a lurker who doesn't stay logged in nor use incognito mode, i have seen "Sorry" page way too often, even when opening the "past" page from the homepage.
truly hope you find a solution that reduces friction for all. personally, it is back to "Sorry" situation for now.
PS: for others facing a similar situation, it all disappears after logging in, which has been the most reliable solution thus far.
xandrius 13 hours ago [-]
Hey dang, don't worry. It's just a site for reading articles and reacting to them.
Enjoy your deserved sleep and if for a couple of hours it's down, so be it.
Thanks for your continued service!
powvans 13 hours ago [-]
100%
Though I will say, HN is a pretty great source of information about major outages like the recent AWS and Cloudflare issues. I had a moment this morning where I thought, oh, is there a larger issue and then, oh, HN is down, huh, the next option is so far down my list that it's going to take me a moment to think of it.
I hope that serves as a testament to how great this site and the community is. Thanks for all your hard work keeping it that way!
derrida 11 hours ago [-]
> huh, the next option is so far down my list that it's going to take me a moment to think of it.
Option 4: take your grab bag with the tcp over IP shortwave radio, sextant and head for pre-cached month supply of food in the hills.
Imustaskforhelp 15 hours ago [-]
I was personally worried if there was some major outage of the whole world or something the first time hackernews didnt work because I didnt expect hackernews to go down but rather, something even more catastrophic than aws going down must happen (because we see major cloud outage posts)
This website had many instances of reports, the last I saw were 52 reports in only a short frame of time, the maximum reports on this are 118 it seems.
> In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.
Its okay I suppose, have you figured out who is crawling hackernews so much tho, was it a ddos attack or an AI company trying to get data, doesn't hackernews support an api and I am sure that there are datasets for it too so Its interesting why they might crawl but we all know the reasons why as they have been discussed here.
neilv 15 hours ago [-]
Maybe it would be fine if ops alerts were silenced during normal US sleeping hours?
HN is important, but unlikely much harm could be done before morning.
(Source: Lost a lot of sleep at one place, enough to realize that sleep interruption and deficit has significant costs.)
_shadi 31 minutes ago [-]
> anti-crawler protections
what type of protections are used on HN? rate-limiting? ip range blacklist?
Rooster61 16 hours ago [-]
No apology needed. We all needed to stop procrastinating anyways :)
tsoukase 15 hours ago [-]
During the last week my IP was banned for unknown reason. Glad to hear it might not be a problem from my side.
dang 12 hours ago [-]
Yes, sorry! This is the problem - we don't want to block legit users, but if we loosen the bolts, we get flooded.
If you browse HN while logged in, that should immunize you against this happening. Also, if it does happen again, you can unban your IP as described at https://news.ycombinator.com/newsfaq.html. But you have to do that from a different IP address, of course.
If those things don't work, email hn@ycombinator.com and we'll get it sorted.
tsoukase 2 hours ago [-]
Thanks. It is so easy to change the IP using mobile that the unbanning is little hassle.
andy_ppp 15 hours ago [-]
I’d love to know more about what running a site like HN involves, would be great to get a write up of what it’s like running something like this at this scale (and what kind of traffic you guys get)!
alwa 15 hours ago [-]
I can’t put my finger on anything within the last decade, but I seem to recall it running in something close to its current form on a single core on a single server for a long time:
I took it as a good reminder that the hard part is the human part: that high-overhead features and UI fripperies are nice but not necessary (or sufficient) to keep a community healthy and vibrant over the decades.
(And on the subject of the human side, if you didn’t catch Anna Wiener’s 2019 profile, it’s here:
The most interesting number is the 1300 submissions because that hasn't grown since 2011 - it just fluctuates. Everything else has been growing more or less linearly for a long time, which is how we like it.
I find that surprising, as 2011-2022 covers an exponential rise in SEO spam and "growth hackers" attempting to drive traffic and links.
Or was 1,300 the number of non-flagged submissions?
tempest_ 15 hours ago [-]
The other reality is that as much as this industry is up its ass about scalability you can run a very very busy site on a single machine now a days.
A lot of people out here designing their blogs like its 1989.
giancarlostoro 15 hours ago [-]
The transparency is deeply appreciated by me and others. We don't pay to keep HN on, so we cannot complain. Thank you and the rest of the team for all you do to give us a corner of the internet that is quite 'different' from the rest of the wild west that is the web.
maxloh 14 hours ago [-]
Frankly, I don't understand why someone would even try to crawl Hacker News.
In a situation like this one, good crisis leadership is essential. dang, HN will help you with tips from vast collected experience (please chip in):
1. Blame: The first thing to do is to point the finger. That doesn't mean analysing the technical issue, which can delay this step and limit your options, but figuring out who is politically easiest to blame. Often, that's the new guy, but outside contractors and vendors without good connections are also a common solution. Even if you are technically responsible for hiring them, you can always push them under the bus with a little skill. This small sacrifice helps unify, focus, and motivate the rest of the team.
2. Emotion: Inject your emotion into the situation and make that the implicit, but indisputable priority. Particularly, outrage and anger - This is completely _____. These people are utterly _____ (I'd use all caps, but that's not allowed on HN). Make sure everyone's attention is over their shoulder, on your emotion, and infect the team with it. Threats are an effective tool here - this is a crisis, and anyone who is calm is not emotionally engaged. Otherwise, they won't care enough about this problem - without you driving them, they probably wouldn't care much at all. Anyway, you don't have time for niceties like empathy or even basic respect.
3. Speed: Respnsiveness to stakeholders is very important. People need answers now. Give them answers they want to hear, outcomes they will be comfortable with. Don't worry if different groups hear different things. Your team will find a way to make it all work - that's their job.
4. Communication: Good communication is essential. Make sure you clearly tell your team what they should be doing; repeat it several times to prevent misunderstanding. Especially people with experience can have minds of their own; keep them on track. The situation is a crisis so you can't take any risks; stay on top of them and everything they do, and give input if you're not certain they are doing exactly what you would be doing.
5. Victimhood: Find a way to turn the tables: Make it about you, and how you're the victim here, and feed the fire with more outrage. With this and outrage, nobody will undermine the team by challenging your ideas or authority, which is the most essential component of a successful outcome. Remember, without you this all falls apart.
Have I missed anything?
yearolinuxdsktp 9 hours ago [-]
Engagement: make sure that every member of the team is either on the incident bridge or has dropped what they are doing to watch you diagnose the problem. The more eyes on the problem, the more awareness of the pain will be absorbed by all. If members need to leave to get food or put children to bed, tell them to order delivery and to ask their spouse to do their job. Demonstrate human touch by allowing them to turn off camera while they are eating.
Comprehensiveness: propose extreme, sweeping solutions, such as a lights-out restart of all services, shutting down all incoming requests, and restoring everything to yesterday's backup. This demonstrates that you are ready to address the problem in a maximally comprehensive way. If someone suggests a config change rollback, or a roll-forward patch, ask them why are gambling company time with localized changes, and ask them why are they willing to gamble company time on technical analysis?
Root Cause Analysis Meeting: spend the entire meeting time rehashing the events, pointing fingers and assigning blame. Be sure to mention how the incident could've been over sooner if you just restarted and rolled back every single thing. Be sure to demonstrate out-of-the-box thinking by discussing unrealistic grandiose solutions. When the time is up, run the meeting over by 30 minutes and force all to stay while realistic solution ideas are finally discussed in overtime. This makes it clear to the team that nothing is more important than this incident's RCA--their time surely is not. If someone asks to tap out to pick their kids up after school, remind them that they are making enough money to call them an Uber.
Alerting: be sure to identify anything remotely resembling leading indicators, and add Critical-level wake-you-up alerts with sensitive thresholds for those indicator. Database exceeding 50% CPU? Critical! Filesystem queue length exceeding 5? Critical! Heap usage over 50%? Critical! 100 errors in one minute on a 100000 requests per minute service? Critical! Single log line indicating DNS resolution failure anywhere in the system? Critical! (What if AWS's DNS is down again?) Service requests rate 10% higher than typical peak? Critical! If anyone objects to such critical alerts, ask them why do they want to be responsible for not preventing the next incident?
showcaseearth 15 hours ago [-]
Short lived and driven by good intentions– all's good. Thanks again for keeping this thing going!
bicepjai 16 hours ago [-]
Even after providing firebase endpoint, crawlers come to the site ?
Bender 15 hours ago [-]
Most crawlers have no concept of what that is. They will follow links to this site and then follow links out of this site even after being told not to [1]. The majority of crawlers follow zero rules, RFC's, etc... The few platforms that do follow standards and rules are akin to a law abiding citizen in Mos Eisley.
[1] - rel="nofollow"
dang 13 hours ago [-]
Oh my god. It's the crawlpocalypse.
busymom0 15 hours ago [-]
Unfortunately, the firebase API is very bad as they even acknowledge that in their github page.
8cvor6j844qw_d6 13 hours ago [-]
> anti-crawler protections
Sometimes I could not open the comment section, receiving a blank page with "... We're sorry" or something along these lines when opening from new private window. It works when opening normally.
Logging in on the private window seems to resolve the issue. Can you take a look on this if possible?
dang 13 hours ago [-]
Best to email your IP address to hn@ycombinator.com so we can see if it's blocked.
nottorp 15 hours ago [-]
Can't speak for others, but I'm sure i'll be pretty fine if no one gets woken up if HN is down...
Of course, they'd better restore service after they wake up naturally, because I need my HN dose. But it's not worth losing sleep over it.
QuantumNomad_ 10 hours ago [-]
> the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users
Was the blocking returning “Sorry.” instead of any page content? A couple of days ago there was a few hours where when I’d go to HN I could load the main page as a non-logged in user. But if I tried to log in I would get “Sorry.” instead. I also got the sorry message if I tried to click on user profiles of other people and a few other pages.
I am assuming that the reason I could see the front page itself and discussions on posts on the front page is that they were in a shared cache for non-logged in users, but that when I clicked on some pages like some random user pages those were not in cache and hit the origin server and it blocked those with “Sorry.” like it did for log-in attempts.
I also tried to go to the unblock IP page, but that one also returned “Sorry.”
For a while I was scratching my head wondering if I had gotten some malware on one of my computers that was aggressively making requests to HN, and that I had become IP banned because of that. Since I think my actual request rate from browsing and commenting should be pretty average. I read HN a lot, but not that much :p
Later in the day, or the next day, things were back to normal and I could log in again. Presumably after those anti-crawler protections had been relaxed again.
irishcoffee 16 hours ago [-]
> The reason I did that is that the anti-crawler protections also unfortunately hit some legit users
How does this happen?
Bender 15 hours ago [-]
How does this happen?
Not the person you are asking. Bot operators have an incentive to make crawlers look as much like a human as possible so they do not get blocked. Some of them fail miserably and some nearly succeed. That makes it trivial to accidentally block a real person. I am personally fine with that given I do not pay for this site and have no SLA or contract with it.
arccy 15 hours ago [-]
some humans also try their best to make themselves look like bots...
slater 14 hours ago [-]
You're absolutely right!
Imustaskforhelp 14 hours ago [-]
beep boop.
ZuoCen_Liu 11 hours ago [-]
lol
pjc50 13 hours ago [-]
Every filter process has false positives and false negatives, especially when crawlers are trying to fake their status.
ohhnoodont 14 hours ago [-]
Last week if you are using a VPN + a browser that limits fingerprinting, you were likely to see error messages accessing HN.
shmeeed 16 hours ago [-]
Looking forward to the post mortem. :)
michelsedgh 16 hours ago [-]
dang
shmeeed 14 hours ago [-]
In my defense, I was commenting at 0 min, since then he made several updates explaining the situation.
dang 13 hours ago [-]
Yes sorry! Normally I put in "[editing - bear with me...]" or some such.
michelsedgh 10 hours ago [-]
I was just trolling, thanks for ur work
stmw 11 hours ago [-]
dang - just to say, we've all done it...
walrus01 12 hours ago [-]
Just out of curiosity, if HN is still running on one physical system, what does a daily or weekly traffic chart look like for the switch port facing it?
echelon 16 hours ago [-]
I didn't realize you were carrying the pager too! Kudos!
malwrar 16 hours ago [-]
I feel such a sense of kinship for anyone who carries a pager, almost 7 years at my current role doing it. Super cool that dang is among our number :)
geocrasher 15 hours ago [-]
Yep, have been on constant "pager duty" for 2+ years, although I have more help now and I get paged 1-3 times a week instead of per night. Still, carry my lappy everywhere I go. Bought an ARM Windows laptop to get that 20hr battery life so I could worry less during my travels. You know, fancy things like going get food or going grocery shopping.
malwrar 13 hours ago [-]
Rough shift, my worst was every other week and my boss prior to hiring me was 24/7 just like you. I just carry a backpack with a few batteries + my work laptop, fortunately only a few really bad stories but hooooo boy me and that backpack have seen some fun times.
wkat4242 10 hours ago [-]
I wish I could still buy a pager where I live :'(
idontwantthis 15 hours ago [-]
Do you carry a literal pager? We use the PagerDuty app.
The AUBMC hospital is definitely using them as well as the paramilitary in that country, at least until recently.
ZuoCen_Liu 11 hours ago [-]
Now, whenever I see a pager, I think of explosions. Haha.
ErroneousBosh 12 hours ago [-]
There's a company in England called "Cascode" who make firefighter alerters. These are really basic "beeper" pagers, which you can program to have a bunch of different tones and LED patterns based on the RIC and Subcode.
I look after several thousand of these across several hundred paging sites.
They're relatively inexpensive (70 quid or so in quantity) and they last about six weeks on a commonly-available AA battery. The batteries go flat enough to trigger the "low battery" beep at about 3am, for some reason. I don't know why.
There's no messaging involved, although the encoders are capable of sending a text string. The message is "get up and get down to the fire station right now", which generally needs no further explanation. POCSAG is unencrypted, so there would be privacy concerns with sending actual incident information in the clear with it.
While we're on the subject of old tech, until BT finally cut the last of them off, we use dialup modems to control the encoders (not dialup internet, just a hundreds-of-miles serial cable) as a backup, and dot-matrix printers to print out a hardcopy message for the crews to pick up.
All very low-tech. All very fixable. All stays working if you don't mess with it.
Encryption is easily doable even with one way pagers. With one way you will lose the perfect forward secrecy option but that's usually ok.
ErroneousBosh 1 hours ago [-]
It's doable but it would be custom firmware and it's not really necessary. Two way paging isn't really worth doing because then you need a massive device with a massive battery, or something that uses uncontrolled mobile phone networks (and generally still has a massive battery, that lasts about a day).
You wouldn't even need particularly good encryption, you'd just need something adequate to stop casual eavesdropping really - "keep them busy for half an hour" would stop people from sniffing the POCSAG traffic and tweeting it, so that people show up at incidents and hang around filming it on their phones.
This incidentally is what a guy in England got arrested for a few years ago, exactly that. It's perfectly legal to listen to and decode pager messages (or any other radio messages), you're just not allowed to pass them on to people or act upon them, and posting them on twitter and then going round to rubberneck at the ongoing incident very much ticks those boxes. As with so many things in the UK, to paraphrase Aleister Crowley, "Don't Be A Dick shall be the whole of the law".
malwrar 13 hours ago [-]
Oh no, I just always hear it termed that way and it captures the “feeling” for me since it feels like a dedicated device. I just just carry a work phone w/ PagerDuty during my shift.
altairprime 16 hours ago [-]
Decades ago I had to write a Perl script to auth to the site for proper downtime checking. Some things never change :) Good luck with the triage.
racl101 15 hours ago [-]
dang!
Elfener 18 hours ago [-]
I got stuck in an infinite loop.
Try opening HN -> it's down, better check HN to see everyone talking about a major website being down -> Try opening HN -> loop
neom 17 hours ago [-]
Yeah me too. Wake up -> HN down -> That's weird, oh well it's usually only down for a few minutes -> I should check if HN is still down -> That's weird, oh well it's usually only down for a few minutes -> I should check if HN is still down -> loop.
That was a few hours ago. I'm glad this loop is broken.
squeefers 17 hours ago [-]
sounds very much like an evil social media dopamine feedback loop. ironic given everyone on HN is so anti social media.... its clearly only bad for kids though i should add, silly of me to exclude such a detail
bee_rider 16 hours ago [-]
HN is obviously social media and it is silly to say otherwise. It is just social media that occasionally has interesting stuff. The SNR is just slightly higher.
ptsd_dalmatian 14 hours ago [-]
I guese developed this addiction on Facebook and now that I don’t use it, I come for methadon shot to HN
neom 17 hours ago [-]
I've been on this internet hit shit since the 90s lil bro, s' all good.
strbean 17 hours ago [-]
I can stop any time I want, I just don't want!
HPsquared 17 hours ago [-]
Sometimes I'll catch myself absentmindedly reopening the browser and checking two or three front pages, seconds after having just checked them and closed the browser.
notachatbot123 16 hours ago [-]
That's a sign of addiction and I highly recommend changing your behaviour towards those pages!
cheschire 14 hours ago [-]
I feel like I need to avoid channelling Bob Saget from Half Baked when I say that is not addiction. That's a habit.
cbracketdash 13 hours ago [-]
There is a noprocrast feature in your settings to specify how long you can stay on for a single session and the frequency at which you can view HN. Super helpful!
Rendello 12 hours ago [-]
Funny, I don't seem to need an outage to get stuck in a HN loop...
mustak_im 16 hours ago [-]
I woke up and was wondering if I’ve just woken up in hell!
Imustaskforhelp 14 hours ago [-]
Yeah, I had assumed something very major must have happened for HN to go down, lmao and I even asked in some linux discord server regarding it asking if there is a major outage as hackernews is down
ErroneousBosh 12 hours ago [-]
Not just me then?
"Shit, HN is down! Hm, I wonder if there's anything about it on HN?"
until stack overflow occurs.
RJIb8RBYxzAMX9u 15 hours ago [-]
HN is how I discover whether other sites are down or not, so it serves a critical function, so of course I check it frequently.
/s
manbitesdog 18 hours ago [-]
TIL I have a "open Hacker News" hand reflex
ectospheno 17 hours ago [-]
I learn more reading the comments here than anywhere else. Thanks everyone for my addiction.
directmusic 16 hours ago [-]
I'm glad I'm not the only one. If I type 'n' into any browser it autocompletes to HN.
embedding-shape 16 hours ago [-]
Save typing hundreds of letters per day, and replace about:newtab with news.ycombinator.com, now you can just do CTRL+T :)
tom1337 15 hours ago [-]
at that moment my productivity would drop to zero
Imustaskforhelp 14 hours ago [-]
mine already has and I dont even have hackernews as my new tab :)
On all fairness though, mine is same for the original comment where just pressing n autocompletes it to https://news.ycombinator.com/
geocrasher 15 hours ago [-]
I had the same thing for Slashdot.org for many, many years. Both the reflex and the browser autocomplete. I still miss the old /. It was like HN + Hackaday + Usenet.
neom 10 hours ago [-]
digg too, till they ruined it...still can't believe they ruined digg.
1shooner 17 hours ago [-]
If you're looking to put the brakes on that, I've used LeechBlock to add a 5-second timer to opening a new HN window (along with other block schedules). The timer even fails if it loses focus, so it really helps slow you down.
echo '127.0.0.1 news.ycombinator.com' | sudo tee -a /etc/hosts
Does the trick as well :) For bonus points (and so you can't workaround it with your phone), do it on your router/switch instead.
You'll still open new tabs and go to HN, but you'll be reminded quickly, and every day can be downtime day \o/ (for you, personally)
squeefers 17 hours ago [-]
so youve got the willpower to do something about it but not enough to just stop doing it?
MikeTheGreat 16 hours ago [-]
To be fair, making a change (particularly changing a habit) takes time. Having something there to remind and nudge you helps make this easier, especially when you're tired, stressed, 'just looking for a short break', etc, etc.
It's like they say: "Your demons will comfort you when no one else will. That's why it's so hard to get rid of them"
1shooner 16 hours ago [-]
Yes.
dwedge 17 hours ago [-]
Have you never suffered from habitual reflexes? I blocked twitter for a while in my hosts file and a dozen times over those first few days I instinctively opened a new tab and typed twitter in
sunrunner 16 hours ago [-]
I deleted the YouTube mobile app a few months ago and I still reflexively reach for the app icon every now and then. Thanks YouTube Shorts.
squeefers 16 hours ago [-]
> I blocked twitter for a while in my hosts file and a dozen times over those first few days I instinctively opened a new tab and typed twitter in?
youd go through that effort when you could have just stopped though.
frikk 16 hours ago [-]
We all admire your absolute mastery of your own habitual reflexes and mind. For the rest of us, there is a daily battle of wits, desires, weakness, and habit.
If I could snap my fingers and break toxic habits and patterns, I would have done so decades ago :)
jstummbillig 18 hours ago [-]
I say. Vibe coded 4 apps once I got past that, on my way to half a billion in ARR already.
wincy 17 hours ago [-]
I’ve turned on no procrast mode and set it to ten minutes per hour. Helped me a lot!
HanClinto 17 hours ago [-]
What are you using to control this?
bee_rider 16 hours ago [-]
It is on your profile, the “noprocrast” dropdown.
randallsquared 16 hours ago [-]
It's a setting available on the page you get from clicking on your own username.
HanClinto 16 hours ago [-]
Whoa. A website that cares about its users enough to _easily support limiting access to itself_?
That's so refreshing in terms of being a user-focused feature, and yet it stands in sharp contrast against today's engagement-hyperfocused climate. I never would have thought to look on a website's own settings page to limit my access to that same website.
I love it, thank you for pointing me to this!
thesurlydev 18 hours ago [-]
Same! Right there with "every day must begin with coffee"
locknitpicker 4 hours ago [-]
> TIL I have a "open Hacker News" hand reflex
You mean it's not your homepage?
AndrewKemendo 18 hours ago [-]
It just reinforces for me that addiction is a human problem not a problem with technology
I know dang basically works
tirelessly to not change the format in order to not induce those addictive patterns
but yet here we all are
chistev 18 hours ago [-]
It's a website with the smartest people in the world. The level of conversations here are unrivaled in internet communities.
It's understandable to be addicted. Lol.
I visit this place multiple times a day.
phantasmish 17 hours ago [-]
99% of social science or political topics and 50% of technical topics here do not… read as smart, and you’d be much better off spending the same time reading the first chapter of a relevant 101-level college textbook.
seizethecheese 17 hours ago [-]
It's entirely possible that this is the smartest place on the internet, but also often dumb. In fact, it seems likely. More of an indictment of the rest of the places on the internet.
squeefers 17 hours ago [-]
> It's entirely possible that this is the smartest place on the internet,
i cant find the link, but there was a post about how to "be nice" and it was a revelation to a worrying amount of "geniuses" on here. bare in mind the sum total of the advice was "be nice, dont be rude"
seizethecheese 13 hours ago [-]
1. niceness and genius are orthogonal
2. your characterization of the article sounds uncharitable
3. my point isn't exactly that this is necessarily the smartest place
ssdspoimdsjvv 16 hours ago [-]
Intelligence has many dimensions.
the_af 15 hours ago [-]
> It's entirely possible that this is the smartest place on the internet, but also often dumb. In fact, it seems likely. More of an indictment of the rest of the places on the internet.
Almost every (non-troll) online community that is relatively peaceful and has some semblance of moderation to remove flamewars thinks of itself as "the best community". Usually as compared to reddit, though if it's on reddit they will compare themselves to some other (hated) sub.
It's a fact of the internet. Every online community thinks of itself as the smartest, more thoughtful, more civilized. HN is no exception.
It goes without saying HN is not the smartest or more thoughtful online community. It's just... ok. Not the worst, not the best. Certainly NOT the place with the smartest people, though some smart people frequent it. As a regular, you can soon figure out HN's unspoken rules, blindspots, and areas where the group opinion is more likely to be accurate.
jrowen 14 hours ago [-]
> It goes without saying HN is not the smartest or more thoughtful online community.
How does that go without saying? Name some others then, compare and contrast. As-is your argument is just posturing.
the_af 13 hours ago [-]
> Name some others then, compare and contrast.
No need, because whether an online community is more thoughtful or smarter than another is very subjective. Almost by definition, HN is not it. Extraordinary claims require extraordinary evidence, and all that. Of course, by internet law, HN (or a subset of its members) considers itself to be the smartest, more thoughtful online community.
There are communities I like better, which are smarter and more thoughtful, but I've no desire to argue with you.
> As-is your argument is just posturing
Nah. Hard pass. Nice try though!
jrowen 3 hours ago [-]
I see you're downvoted, it wasn't me. I wasn't making any claim, you're making claims and disparaging remarks that you won't substantiate.
steve_adams_86 17 hours ago [-]
> the smartest people in the world
But also, people like me. Be careful what you choose to believe on this website
fwip 18 hours ago [-]
It's really not 'the smartest people.' It's people interested in tech, and often in making-a-lot-of-money-in-tech. It does have a lot of people with significant industry experience, which is cool.
worksonmine 17 hours ago [-]
> It's really not 'the smartest people.'
This was especially obvious during Covid, I even stopped visiting because the comment section was so crazy.
the_af 17 hours ago [-]
> It's a website with the smartest people in the world.
Nice joke!
At least, I hope it was a joke...
17 hours ago [-]
18 hours ago [-]
linhns 17 hours ago [-]
Also the level of flak is unrivaled.
krapp 13 hours ago [-]
Any professional forum or technical subreddit with good moderation and gatekeeping blows Hacker News out of the water any day of the week.
andrepd 16 hours ago [-]
Poe's Law for a parody of the self-important sv techbro
the_af 15 hours ago [-]
It's a testament to Poe's Law that I genuinely cannot tell if the OP was being funny or not.
mrguyorama 13 hours ago [-]
There's a reason why HN of all places has such a terrible record of handling actual sarcasm and telling it apart from genuine belief and that reason is NOT that "HN is really smart" lol
the_af 13 hours ago [-]
Agreed! I think HN is average, and its userbase think themselves smarter than they really are.
... but I still cannot tell if the original commenter was sarcastic or not! ;)
dzink 17 hours ago [-]
This one is at least healthy-ish for the mind. I’d much rather hacker news than any other news. Social Media is an emotional rage-bait cesspool these days. If it’s not for Hacker News those of us who abstain from the rest would be living in the dark.
PurpleRamen 17 hours ago [-]
But, would the addiction become worse if HN changed, or would there be a point where they could cure it?
ZuoCen_Liu 11 hours ago [-]
Perhaps this is the role of HN - at least it still allows us to reflect.
yigithan 6 hours ago [-]
I stopped using google.com for Internet access check, I now use HN.
cbracketdash 13 hours ago [-]
There is a noprocrast feature in your settings to specify how long you can stay on for a single session and the frequency at which you can view HN. Super helpful!
lysace 17 hours ago [-]
⌘-T, N, <RET>
Did it like 5 times during that 1h-ish outage. :(
ChrisMarshallNY 17 hours ago [-]
So do I, but it was such a shock that I just passed out, and when I woke up, it was back up.
ZuoCen_Liu 11 hours ago [-]
Admirable~
ErroneousBosh 12 hours ago [-]
Do you log into things and reflexively type "ls", too?
kevin061 18 hours ago [-]
I did not know how addicted I was to HN until today lol
Yes, and I'm a little ashamed to admit my morning routine wasn't the same without it.
fedreg 18 hours ago [-]
This was more impactful to my day than the last AWS and CloudFlare outages...
messe 18 hours ago [-]
At least during those outages I could procrastinate on HN.
the_arun 17 hours ago [-]
I felt like changing HN down page to show top 30 posts from this week before or after the generic message.
al_borland 18 hours ago [-]
Is this still a valid account for HN status? It says it’s the official one, but with the changes at Twitter to no longer show chronological feeds (at least for users that aren’t logged in), it’s rather useless. The top 5 listed post (for me) are seemingly random from 2014 - 2022.
https://xcancel.com/HNStatus only uses chronological ordering (after any pinned tweets) and that has the last message 12 Dec 2023.
(Basically whenever you see an x.com link just change it to xcancel.com and avoid the nonsense.)
FuriouslyAdrift 17 hours ago [-]
Only way I have figured out how to to change the "Following" sort order back to chronological is from the mobile app: click the down arrow on the "Following" tab. Change the sort from "popular" to "most recent."
Seems to reset it on the web view, too.
al_borland 16 hours ago [-]
It sounds like this would only work for logged in users.
zipy124 18 hours ago [-]
https://hn.hund.io/ Is a status page, no idea if official or not, but it didn't register here for some reason.
I didn't read the post text, it's identified there haha, my bad! I wish the text post text wasn't grey, I gloss over it too easily.
laCour 17 hours ago [-]
This was monitoring the unauthenticated news page, which is why it didn't catch it. It now monitors authentication as well. It is not official, and was made by a co-founder years ago.
This site said HN was fine and green the entire time it was down.
liampulles 17 hours ago [-]
Smart. Have to use that error budget before year end...
dylan604 17 hours ago [-]
I always hated the late use-it-or-loose-it at the end of the year where you end up buying the things that were denied requests from earlier in the year. You just cost me half a year of using the damn thing.
wiredpancake 12 hours ago [-]
[dead]
ramonga 1 hours ago [-]
when I realised the api still worked I put this together https://hackernewsx.com just to realise it will just work if I cleared my site data.
Interestingly it stayed up if you weren't logged in.
Izkata 17 hours ago [-]
Not completely, I'm not logged in on my work laptop and it was only working some of the time (and not like some pages were cached and some weren't, I was refreshing the same page and sometimes it worked and sometimes not).
cess11 17 hours ago [-]
That's how I concluded that it wasn't a ban on my account but rather more serious.
jedberg 18 hours ago [-]
If you aren't logged in you get a cached version from the CDN/cache. Reddit works the same way.
bryanrasmussen 18 hours ago [-]
also went down if you went to login, and people's individual pages were also down. So as far as I saw the front page was up as long as you were not logged in, however I'm not sure if that wasn't just luck of the draw, I had one experience where it looked like maybe the front page was sometimes down for not logged in users as well.
on edit: ok others pointed out it was cached pages I saw. explains it.
smallerize 18 hours ago [-]
That only worked for a while, eventually I couldn't load comment pages even logged out.
davnicwil 18 hours ago [-]
that'll be because it's served from cache when you're not logged in
wavemode 18 hours ago [-]
When it was down my thought was "damnit, I'll actually have a productive workday now."
tzs 18 hours ago [-]
Next time you can avoid that fate by opening HN in a private browsing (or whatever your browser calls its equivalent) window. This outage, like the vast majority of HN outages, only affected logged in requests.
I suppose you could also just clear your HN cookies in regular browsing window, but then when they fix it you'd have to log in again.
dpoloncsak 17 hours ago [-]
Huh. Dunno why, but when it failed on Firefox I tried Chrome, and it worked. I wrote it off as a Mozilla issue, but this would better explain that I think
neom 17 hours ago [-]
Is it my imagination or did they used to automatically serve you a logged out page when it was down?
ycombinator_acc 14 hours ago [-]
I couldn't access it in private or regular.
dwa3592 18 hours ago [-]
hahaha
esafak 18 hours ago [-]
A lot of the outage indicators failed. Someone needs to create an outage indicator reliability dashboard.
Nextgrid 18 hours ago [-]
A lot of them got fooled by the caching; pages for signed-out users are cached heavily and those kept returning successful responses even if the actual backend server was down.
I was quiet surprised really. HN almost never goes down.
Aachen 17 hours ago [-]
You don't visit enough my friend :)
It's down about 8.4 minutes per week. On 26% of days it doesn't work at least once, and on 12% of days it has more than one consecutive failed check. The longest uptime streak was 24 days
I've been keeping track since exactly 2 years (to the day!) because I was surprised that it seemed briefly down for me on a daily basis. Was I getting unlucky and hitting it every time, or was it just down very often? Nobody posted anything so I started answering the question for myself :p
I've been meaning to post the tracker to HN but there's a pesky bug I want to fix: the "is it currently down" stat. I don't know how this is beyond me but something in the code bugs out. So this is my first time posting about it
ayewo 15 hours ago [-]
Would love to see it when you do get round to posting it :)
dustfinger 15 hours ago [-]
It would be a nice feature if hackernews had a simple news.ycombinator.com/status page that I could trust. I don't like loading those some-sketchy-domain/status-right-now sites.
mmooss 14 hours ago [-]
> news.ycombinator.com/status page
That's not so useful when news.ycombinator.com is having problems.
dustfinger 14 hours ago [-]
True. how about https://ycombinator-status.com or https://status.ycombinator.com. They can drop that link on their official ycombinator.com site just so it is easy to tell that it is legit when they are not down. They can maintain the status for all of their official sites there. Well, with the exception of ycombinator-status.com of course. I guess they can post that status on some-sketchy-domain/status-right-now - LOL!
Maybe ycombinator does have an official status page somewhere, but it is not easy to find if that is the case.
danielfalbo 17 hours ago [-]
As soon as I noticed it was down I came to hacker news to post about it, but...
I was able to view the site without being signed in (i.e. private window) but any browser I was logged into wouldn't load.
I'm sure it's a coincidence but it started working again shortly after emailing hn@ycombinator.com
whynotmaybe 16 hours ago [-]
I Still found it funny that we have downdetector, downdetectordowndetector, downdetectordowndetectordowndetector,... (I lost count) but downdetector doesn't detect HN's status.
chistev 18 hours ago [-]
It was the first time since I started using this website (August last year) that it was down.
I'm still impressed nonetheless.
I'd like to know what caused the outage and how it could have been prevented, for learning purposes.
I got irrationally angry when it refused to load the website
imvetri 18 hours ago [-]
you could refuse to be angry to mock it
DonHopkins 18 hours ago [-]
You handle your irrational anger much better than whatevermrfukz who keeps pooping his pants.
fatty_patty89 18 hours ago [-]
Interesting that you mention that, how do you keep track of his comments? Some sort of plugin/scraper?
Jtsummers 17 hours ago [-]
> how do you keep track of his comments?
You can just look at them, turn on showdead in your profile and you'll see a bunch of flag-killed comments in this discussion by whatevermrfukz. No need for a plugin or scraper.
fatty_patty89 17 hours ago [-]
thank you
zelphirkalt 18 hours ago [-]
Yep "having trouble serving my request" or so.
10 hours ago [-]
PurpleRamen 17 hours ago [-]
In other words, productivity in tech skyrocketed for hours..though it seems some work was flavoured with irrational anger.
khobragade 16 hours ago [-]
Man if I had a coin every time I email the mods and this thing goes down, I'd strangely have two coins.
Kim_Bruning 17 hours ago [-]
In with the rest, yes, and my first thought is always "I have an internet outage" when HN is down. :-P
numpad0 18 hours ago [-]
Could have been accidental flagging of sorts. Didn't work on PC for few minutes while showing fine on phone.
sangeeth96 17 hours ago [-]
Thought it had something to do with some model updates, like Gemini Flash 3 for a moment.
arm32 17 hours ago [-]
The "site status" apps are all smoke and mirrors, how unreliable.
oidar 17 hours ago [-]
What was the longest that HN has been down? I feel like this is up there.
pedro380085 18 hours ago [-]
Yes, I got an error message, but I cleared my cookies and was able to access.
dvaun 17 hours ago [-]
Will we get a post mortem?
18 hours ago [-]
voxleone 18 hours ago [-]
Yes, it' been out for me too, southern hemisphere, GMT -03
neurolesudiste 17 hours ago [-]
Hi,
Got an error an hour ago on phone, not loged.
Anyway, glad to see you back.
Paris 1812.
Cheers from France.
18 hours ago [-]
rustaceanU32 17 hours ago [-]
I'll admit this ruined my morning
arbirk 17 hours ago [-]
We need a serious post mortem for this
neurolesudiste 18 hours ago [-]
Yup, France here , Paris 1809.
HN was down about an hour ago.
Glad to see it back !
Cheers.
silverpiranha 18 hours ago [-]
Yes, was down for me too this morning
markus_zhang 18 hours ago [-]
Yes was down for me. Ontario, Canada.
gepiti 17 hours ago [-]
Athens, Greece yes it was down.
wek 18 hours ago [-]
Yes, it was down for a few hours
gaigalas 17 hours ago [-]
I got confused by the "minutes ago" thing.
Working with full dates in the HTML and doing a tiny JavaScript that calculates the "minutes ago" would actually be a neat improvement.
thatgerhard 18 hours ago [-]
Thought for a second I got banned for something lol
elxr 17 hours ago [-]
I thought I was being rate-limited for opening posts too fast, which has happened before.
After more than an hour I thought, "wow this is pretty harsh" and "so much of my exposure to learning things is directly tied to HN posts". I was lost lol.
verzali 17 hours ago [-]
Me too and I was wondering what I did!
Aachen 17 hours ago [-]
Your comment history doesn't sound IP-ban-worthy. Is there an alt that has you worried? :P
agumonkey 17 hours ago [-]
hn applied ycombinator too strictly
bossyTeacher 16 hours ago [-]
I have never before seen this website down.
SilverElfin 17 hours ago [-]
I tried to refresh an embarrassing amount of times
russellbeattie 17 hours ago [-]
HN being down makes you start wondering about the differences between routine, addiction, compulsion, and habit.
khaledh 18 hours ago [-]
It was down if you tried to access it while authenticated (i.e. you have a cookie). It was loading fine for unauthenticated sessions (e.g. incognito).
rzerowan 18 hours ago [-]
Yes
busymom0 18 hours ago [-]
I run the app called HACK and received user emails that the HN website was down.
maverwa 18 hours ago [-]
Thank you for HACK. I love it!
spooneybarger 18 hours ago [-]
Yes
ProofHouse 18 hours ago [-]
I wake up, and look at HackerNews lol. This morning sucked hahaha
bennydog224 18 hours ago [-]
Yes, I had to touch grass this morning.
complianceowl 14 hours ago [-]
I was convinced that I got banned lol.
FergusArgyll 18 hours ago [-]
Yeah & chatgpt was trying to gaslight me - claiming it was my fault. Happy to put that bum in its place...
chistev 17 hours ago [-]
You use ChatGPT to know if a site is down?
laCour 17 hours ago [-]
I'm with Hund. Our hn.hund.io page did not catch this because it was requesting the cached, unauthenticated page. It now monitors authentication as well.
joncrane 17 hours ago [-]
Thank you. I was thinking myself or my corporate IP was shadowbanned
sammy2255 14 hours ago [-]
You should add a graph of visitors per-minute for the status page for the past 24 hours or so. Would really help for situations like this
jonahx 17 hours ago [-]
Is this a mistake by hund, or the configuration of hund by HN?
laCour 17 hours ago [-]
Mistake on our part (Hund) for not monitoring authentication. This page is unofficial and was made by a co-founder several years ago.
16 hours ago [-]
Cheer2171 16 hours ago [-]
[flagged]
dang 13 hours ago [-]
Yikes, you can't do this here.
We've banned this account for repeatedly breaking the site guidelines and ignoring our requests to stop.
If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
sentrysapper 15 hours ago [-]
What a rude thing to post. Hund, don't listen to this entitled nonsense. There is a reason it's called human error. Companies 100x your size and 10000x your revenue like AWS, Microsoft, CloudFlare, CrowdStrike can't figure out how to accurately provide status dashboards. At least you took the time to explain your mistakes. If anything you got another supporter for your honesty.
seanw444 15 hours ago [-]
This just in: people make mistakes on occasion.
pcdoodle 18 hours ago [-]
[dead]
gjsman-1000 18 hours ago [-]
[flagged]
Aachen 17 hours ago [-]
Keep in mind that votes aren't supposed to be about whether you agree or disagree, but whether a comment adds substance to the thread. A good question that makes a wrong assumption can be worth upvoting; a correct but fairly irrelevant statement about politics not (to give a relatively obvious example)
Being "voted to -2" doesn't necessarily mean you were wrong (it often correlates though). People might just think it wasn't relevant in whatever context you posted it in
I often find it hard to tell what makes people think something I write is not helpful (or sometimes also a comment someone else made) and thus appreciate comments that clarify constructively. It can also help to ask for clarification if you're particularly surprised about the votes on a given post
Eikon 15 hours ago [-]
> Keep in mind that votes aren't supposed to be about whether you agree or disagree
That’s not what happens in practice.
abujazar 18 hours ago [-]
Unless the downtime was caused by something Cloudflare would've prevented, this downtime would've happened regardless of being behind Cloudflare. Cloudflare adds another single point of failure.
null_deref 18 hours ago [-]
Could this be a self-inflicted bug? In that case, the broader point still stands: cloud providers can cause outages that are outside your direct realm of responsibility.
gjsman-1000 18 hours ago [-]
Your VPS server and your data center and the ISP your data center uses and the AS system your ISP uses all can cause outages outside your direct realm of responsibility.
Jtsummers 18 hours ago [-]
Complaining about downvotes usually gets you more downvotes, so the response here isn't really that surprising.
Jeremy1026 16 hours ago [-]
You're probably being downvoted today for your "I was smarter than you" tone.
DonHopkins 18 hours ago [-]
It's absolutely irresistible downvoting people who preemptively complain about being downvoted like you do. It really made my day. Post another complaint so I can do it again please! It's not knee jerk when you explicitly ask for it, by leading with a complaint about downvoting, instead of just making your point and letting it fall or rise on its own merits. You're the one who put the idea of downvoting you into my head in the first place.
18 hours ago [-]
whatevermrfukz 18 hours ago [-]
[flagged]
whatevermrfukz 18 hours ago [-]
[flagged]
whatevermrfukz 18 hours ago [-]
[flagged]
DonHopkins 18 hours ago [-]
I remember my first beer.
null_deref 18 hours ago [-]
That Markov chain spiraled out of control
rolandog 18 hours ago [-]
Perhaps Grok was not trained to go so much time with 0 responses due to the downtime?
whatevermrfukz 18 hours ago [-]
[flagged]
Rendered at 10:41:02 GMT+0000 (Coordinated Universal Time) with Vercel.
(The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)
In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.
I'll add more as we find out more, but it probably won't be till later this afternoon PST.
We all knew that but I haven't seen any confirmation before this.
I think you're confusing popularity with criticality. I'm sure everyone in here can withstand a few hours without browsing the page.
It's dang's baby at this point, and this is a good thing, as long as HN doesn't affect his life in ways he doesn't want.
Get a grip and go touch some grass. Even FANGs understand the concept of business hours SEV.
However, when something I care about crashes and burns once in a blue moon, I make sure to put the fire out, at least to make it survive till regular hours. Things I care about can be both business and personal, and nobody bugs me for them.
Maybe we shouldn't make any assumptions about people we don't personally know, while we are at it.
I hope it doesn't change (much).
So don’t beat yourself up please.
When I worked for “SaaS unicorn” we typically had multiple levels of escalation, and acknowledging would have done nothing because the alarm would continue firing until fixed. Not sure what’s changed in 15 years of ops, I had assumed it would be better now- I can’t imagine silencing an alert totally by acknowledging it- if its still occurring.
I’m totally fine with how you handled it, if anything I am thankful. But that seems to be a system I would improve if I had the time.
“mute” is different than “resolve” to me, and both should exist. (Where mute is an acknowledgement of an issue as ongoing.)
Not to say that I don't procrastinate or waste time doing other nonsense. I can definitely spend a lot of time reading HN comments, as I'm doing right now.
Anyway,anyone who finds themselves with a problem with HN should try that out :)
(Might be wise though to have PagerDuty configured to re-alert if the outage persists.)
I'm pretty happy with how it's developing—the trendline is promising—but not ready to rely on it in prod yet.
it is a shame that it needs to be this way. as a lurker who doesn't stay logged in nor use incognito mode, i have seen "Sorry" page way too often, even when opening the "past" page from the homepage.
truly hope you find a solution that reduces friction for all. personally, it is back to "Sorry" situation for now.
PS: for others facing a similar situation, it all disappears after logging in, which has been the most reliable solution thus far.
Enjoy your deserved sleep and if for a couple of hours it's down, so be it.
Thanks for your continued service!
Though I will say, HN is a pretty great source of information about major outages like the recent AWS and Cloudflare issues. I had a moment this morning where I thought, oh, is there a larger issue and then, oh, HN is down, huh, the next option is so far down my list that it's going to take me a moment to think of it.
I hope that serves as a testament to how great this site and the community is. Thanks for all your hard work keeping it that way!
Option 4: take your grab bag with the tcp over IP shortwave radio, sextant and head for pre-cached month supply of food in the hills.
https://downforeveryoneorjustme.com/hacker-news
This website had many instances of reports, the last I saw were 52 reports in only a short frame of time, the maximum reports on this are 118 it seems.
> In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.
Its okay I suppose, have you figured out who is crawling hackernews so much tho, was it a ddos attack or an AI company trying to get data, doesn't hackernews support an api and I am sure that there are datasets for it too so Its interesting why they might crawl but we all know the reasons why as they have been discussed here.
HN is important, but unlikely much harm could be done before morning.
(Source: Lost a lot of sleep at one place, enough to realize that sleep interruption and deficit has significant costs.)
what type of protections are used on HN? rate-limiting? ip range blacklist?
If you browse HN while logged in, that should immunize you against this happening. Also, if it does happen again, you can unban your IP as described at https://news.ycombinator.com/newsfaq.html. But you have to do that from a different IP address, of course.
If those things don't work, email hn@ycombinator.com and we'll get it sorted.
https://news.ycombinator.com/item?id=5229522
Re: traffic, dang said (2022):
https://news.ycombinator.com/item?id=33454140
I took it as a good reminder that the hard part is the human part: that high-overhead features and UI fripperies are nice but not necessary (or sufficient) to keep a community healthy and vibrant over the decades.
(And on the subject of the human side, if you didn’t catch Anna Wiener’s 2019 profile, it’s here:
https://www.newyorker.com/news/letter-from-silicon-valley/th... )
The most interesting number is the 1300 submissions because that hasn't grown since 2011 - it just fluctuates. Everything else has been growing more or less linearly for a long time, which is how we like it.
I find that surprising, as 2011-2022 covers an exponential rise in SEO spam and "growth hackers" attempting to drive traffic and links.
Or was 1,300 the number of non-flagged submissions?
A lot of people out here designing their blogs like its 1989.
There is an official dump which doesn't even require parsing HTML at all: https://console.cloud.google.com/marketplace/details/y-combi...
https://www.youtube.com/watch?v=Sbpl3ywNlpA#t=56s
1. Blame: The first thing to do is to point the finger. That doesn't mean analysing the technical issue, which can delay this step and limit your options, but figuring out who is politically easiest to blame. Often, that's the new guy, but outside contractors and vendors without good connections are also a common solution. Even if you are technically responsible for hiring them, you can always push them under the bus with a little skill. This small sacrifice helps unify, focus, and motivate the rest of the team.
2. Emotion: Inject your emotion into the situation and make that the implicit, but indisputable priority. Particularly, outrage and anger - This is completely _____. These people are utterly _____ (I'd use all caps, but that's not allowed on HN). Make sure everyone's attention is over their shoulder, on your emotion, and infect the team with it. Threats are an effective tool here - this is a crisis, and anyone who is calm is not emotionally engaged. Otherwise, they won't care enough about this problem - without you driving them, they probably wouldn't care much at all. Anyway, you don't have time for niceties like empathy or even basic respect.
3. Speed: Respnsiveness to stakeholders is very important. People need answers now. Give them answers they want to hear, outcomes they will be comfortable with. Don't worry if different groups hear different things. Your team will find a way to make it all work - that's their job.
4. Communication: Good communication is essential. Make sure you clearly tell your team what they should be doing; repeat it several times to prevent misunderstanding. Especially people with experience can have minds of their own; keep them on track. The situation is a crisis so you can't take any risks; stay on top of them and everything they do, and give input if you're not certain they are doing exactly what you would be doing.
5. Victimhood: Find a way to turn the tables: Make it about you, and how you're the victim here, and feed the fire with more outrage. With this and outrage, nobody will undermine the team by challenging your ideas or authority, which is the most essential component of a successful outcome. Remember, without you this all falls apart.
Have I missed anything?
Comprehensiveness: propose extreme, sweeping solutions, such as a lights-out restart of all services, shutting down all incoming requests, and restoring everything to yesterday's backup. This demonstrates that you are ready to address the problem in a maximally comprehensive way. If someone suggests a config change rollback, or a roll-forward patch, ask them why are gambling company time with localized changes, and ask them why are they willing to gamble company time on technical analysis?
Root Cause Analysis Meeting: spend the entire meeting time rehashing the events, pointing fingers and assigning blame. Be sure to mention how the incident could've been over sooner if you just restarted and rolled back every single thing. Be sure to demonstrate out-of-the-box thinking by discussing unrealistic grandiose solutions. When the time is up, run the meeting over by 30 minutes and force all to stay while realistic solution ideas are finally discussed in overtime. This makes it clear to the team that nothing is more important than this incident's RCA--their time surely is not. If someone asks to tap out to pick their kids up after school, remind them that they are making enough money to call them an Uber.
Alerting: be sure to identify anything remotely resembling leading indicators, and add Critical-level wake-you-up alerts with sensitive thresholds for those indicator. Database exceeding 50% CPU? Critical! Filesystem queue length exceeding 5? Critical! Heap usage over 50%? Critical! 100 errors in one minute on a 100000 requests per minute service? Critical! Single log line indicating DNS resolution failure anywhere in the system? Critical! (What if AWS's DNS is down again?) Service requests rate 10% higher than typical peak? Critical! If anyone objects to such critical alerts, ask them why do they want to be responsible for not preventing the next incident?
[1] - rel="nofollow"
Sometimes I could not open the comment section, receiving a blank page with "... We're sorry" or something along these lines when opening from new private window. It works when opening normally.
Logging in on the private window seems to resolve the issue. Can you take a look on this if possible?
Of course, they'd better restore service after they wake up naturally, because I need my HN dose. But it's not worth losing sleep over it.
Was the blocking returning “Sorry.” instead of any page content? A couple of days ago there was a few hours where when I’d go to HN I could load the main page as a non-logged in user. But if I tried to log in I would get “Sorry.” instead. I also got the sorry message if I tried to click on user profiles of other people and a few other pages.
I am assuming that the reason I could see the front page itself and discussions on posts on the front page is that they were in a shared cache for non-logged in users, but that when I clicked on some pages like some random user pages those were not in cache and hit the origin server and it blocked those with “Sorry.” like it did for log-in attempts.
I also tried to go to the unblock IP page, but that one also returned “Sorry.”
For a while I was scratching my head wondering if I had gotten some malware on one of my computers that was aggressively making requests to HN, and that I had become IP banned because of that. Since I think my actual request rate from browsing and commenting should be pretty average. I read HN a lot, but not that much :p
Later in the day, or the next day, things were back to normal and I could log in again. Presumably after those anti-crawler protections had been relaxed again.
How does this happen?
Not the person you are asking. Bot operators have an incentive to make crawlers look as much like a human as possible so they do not get blocked. Some of them fail miserably and some nearly succeed. That makes it trivial to accidentally block a real person. I am personally fine with that given I do not pay for this site and have no SLA or contract with it.
My pager noise: https://www.soundjay.com/transportation/sounds/train-crossin...
That will not only wake the dead, it'll wake me no matter how asleep I am.
I used to work on Motorola Minitor 5 pagers. Looks like they recently released their newest model, the Minitor 7
I wonder if pagers are still used in hospitals? I imagine so
I look after several thousand of these across several hundred paging sites.
They're relatively inexpensive (70 quid or so in quantity) and they last about six weeks on a commonly-available AA battery. The batteries go flat enough to trigger the "low battery" beep at about 3am, for some reason. I don't know why.
There's no messaging involved, although the encoders are capable of sending a text string. The message is "get up and get down to the fire station right now", which generally needs no further explanation. POCSAG is unencrypted, so there would be privacy concerns with sending actual incident information in the clear with it.
While we're on the subject of old tech, until BT finally cut the last of them off, we use dialup modems to control the encoders (not dialup internet, just a hundreds-of-miles serial cable) as a backup, and dot-matrix printers to print out a hardcopy message for the crews to pick up.
All very low-tech. All very fixable. All stays working if you don't mess with it.
https://cascode.co.uk/products/2ar2-and-2ar3/
You wouldn't even need particularly good encryption, you'd just need something adequate to stop casual eavesdropping really - "keep them busy for half an hour" would stop people from sniffing the POCSAG traffic and tweeting it, so that people show up at incidents and hang around filming it on their phones.
This incidentally is what a guy in England got arrested for a few years ago, exactly that. It's perfectly legal to listen to and decode pager messages (or any other radio messages), you're just not allowed to pass them on to people or act upon them, and posting them on twitter and then going round to rubberneck at the ongoing incident very much ticks those boxes. As with so many things in the UK, to paraphrase Aleister Crowley, "Don't Be A Dick shall be the whole of the law".
Try opening HN -> it's down, better check HN to see everyone talking about a major website being down -> Try opening HN -> loop
That was a few hours ago. I'm glad this loop is broken.
"Shit, HN is down! Hm, I wonder if there's anything about it on HN?"
until stack overflow occurs.
/s
On all fairness though, mine is same for the original comment where just pressing n autocompletes it to https://news.ycombinator.com/
https://www.proginosko.com/leechblock/
You'll still open new tabs and go to HN, but you'll be reminded quickly, and every day can be downtime day \o/ (for you, personally)
It's like they say: "Your demons will comfort you when no one else will. That's why it's so hard to get rid of them"
youd go through that effort when you could have just stopped though.
If I could snap my fingers and break toxic habits and patterns, I would have done so decades ago :)
That's so refreshing in terms of being a user-focused feature, and yet it stands in sharp contrast against today's engagement-hyperfocused climate. I never would have thought to look on a website's own settings page to limit my access to that same website.
I love it, thank you for pointing me to this!
You mean it's not your homepage?
I know dang basically works tirelessly to not change the format in order to not induce those addictive patterns
but yet here we all are
It's understandable to be addicted. Lol.
I visit this place multiple times a day.
i cant find the link, but there was a post about how to "be nice" and it was a revelation to a worrying amount of "geniuses" on here. bare in mind the sum total of the advice was "be nice, dont be rude"
2. your characterization of the article sounds uncharitable
3. my point isn't exactly that this is necessarily the smartest place
Almost every (non-troll) online community that is relatively peaceful and has some semblance of moderation to remove flamewars thinks of itself as "the best community". Usually as compared to reddit, though if it's on reddit they will compare themselves to some other (hated) sub.
It's a fact of the internet. Every online community thinks of itself as the smartest, more thoughtful, more civilized. HN is no exception.
It goes without saying HN is not the smartest or more thoughtful online community. It's just... ok. Not the worst, not the best. Certainly NOT the place with the smartest people, though some smart people frequent it. As a regular, you can soon figure out HN's unspoken rules, blindspots, and areas where the group opinion is more likely to be accurate.
How does that go without saying? Name some others then, compare and contrast. As-is your argument is just posturing.
No need, because whether an online community is more thoughtful or smarter than another is very subjective. Almost by definition, HN is not it. Extraordinary claims require extraordinary evidence, and all that. Of course, by internet law, HN (or a subset of its members) considers itself to be the smartest, more thoughtful online community.
There are communities I like better, which are smarter and more thoughtful, but I've no desire to argue with you.
> As-is your argument is just posturing
Nah. Hard pass. Nice try though!
But also, people like me. Be careful what you choose to believe on this website
This was especially obvious during Covid, I even stopped visiting because the comment section was so crazy.
Nice joke!
At least, I hope it was a joke...
... but I still cannot tell if the original commenter was sarcastic or not! ;)
Did it like 5 times during that 1h-ish outage. :(
https://x.com/HNStatus
Is there a better place to check, beyond a basic down detector that may provide more insight or signal that the outage is acknowledged?
(Basically whenever you see an x.com link just change it to xcancel.com and avoid the nonsense.)
Seems to reset it on the web view, too.
I didn't read the post text, it's identified there haha, my bad! I wish the text post text wasn't grey, I gloss over it too easily.
https://hackernews.onlineornot.com/incidents/yaz-eOJeARBL
https://downforeveryoneorjustme.com/hacker-news
Strangely, nothing from the statuspal, which is the first google result
https://hacker-news.statuspal.io/
on edit: ok others pointed out it was cached pages I saw. explains it.
I suppose you could also just clear your HN cookies in regular browsing window, but then when they fix it you'd have to log in again.
I believe it's because they accept user reports.
https://x.com/paulg/status/1953289830982664236?s=46
It's down about 8.4 minutes per week. On 26% of days it doesn't work at least once, and on 12% of days it has more than one consecutive failed check. The longest uptime streak was 24 days
I've been keeping track since exactly 2 years (to the day!) because I was surprised that it seemed briefly down for me on a daily basis. Was I getting unlucky and hitting it every time, or was it just down very often? Nobody posted anything so I started answering the question for myself :p
I've been meaning to post the tracker to HN but there's a pesky bug I want to fix: the "is it currently down" stat. I don't know how this is beyond me but something in the code bugs out. So this is my first time posting about it
That's not so useful when news.ycombinator.com is having problems.
Maybe ycombinator does have an official status page somewhere, but it is not easy to find if that is the case.
It did work without being logged on. The auth service appeared to be down as the log in attempt (just showing the page) failed.
It's not that much different from HN, come to think of it.
(ha, ha)
I'm sure it's a coincidence but it started working again shortly after emailing hn@ycombinator.com
I'm still impressed nonetheless.
I'd like to know what caused the outage and how it could have been prevented, for learning purposes.
You can just look at them, turn on showdead in your profile and you'll see a bunch of flag-killed comments in this discussion by whatevermrfukz. No need for a plugin or scraper.
Anyway, glad to see you back.
Paris 1812.
Cheers from France.
HN was down about an hour ago.
Glad to see it back !
Cheers.
Working with full dates in the HTML and doing a tiny JavaScript that calculates the "minutes ago" would actually be a neat improvement.
After more than an hour I thought, "wow this is pretty harsh" and "so much of my exposure to learning things is directly tied to HN posts". I was lost lol.
We've banned this account for repeatedly breaking the site guidelines and ignoring our requests to stop.
If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
Being "voted to -2" doesn't necessarily mean you were wrong (it often correlates though). People might just think it wasn't relevant in whatever context you posted it in
I often find it hard to tell what makes people think something I write is not helpful (or sometimes also a comment someone else made) and thus appreciate comments that clarify constructively. It can also help to ask for clarification if you're particularly surprised about the votes on a given post
That’s not what happens in practice.