Spent 5 minutes debugging why my builds were failing... then recalled github is not google when it comes to availability... checked status.github.com ... and voila.. there it is.
I'm surprised people still rely so much on GitHub. Anything I want to (have to*) be able to deploy on a moments notice, for hotfixes and what not, was moved away from Github like a year ago, because of the horrible uptime. Did others not get the memo yet?
It's not getting better, if anything it's getting worse. Best time to get off GitHub was yesterday, the second-best day to get off is today. Tangled, Codeberg or self-hosted Forgejo (my approach) or Gitea are all good alternative solutions here.
It's free and you can get it up and running pretty much instantaneously with a coding agent. That's it.
They really should be partitioning their infrastructure so that their paying customers are not in the blast radius of outages from the legions of repos running their 700 test vibecoded regression suite every time they change a config file.
I’m sure they’re working on that, but that takes time. It definitely doesn’t happen in the relative “overnight” that this new influx of code spam spun up.
>> Anything I want to (have to*) be able to deploy on a moments notice, for hotfixes and what not, was moved away from Github like a year ago, because of the horrible uptime. Did others not get the memo yet?
This misses the point in a bad way. You shouldn't have a single deploy funnel to begin with. You should be able to do deploys from multiple places.
Sure, and backups should be replicated over 3 mediums, and you shouldn't deploy bugs into production, and...
Lots of things we should do, only a few we actually have time to fix. If I were choosing between "Migrating away from GitHub" and "Adding another way to deploy in case GitHub is down", I'd take advantage and do the first, because you'll end up having to replicate SCM, build infrastructure and so on anyway, why not do it properly?
What do you think parent is trying to say, except that this HN comment section looks like reddit? I'm trying my hardest to apply charitable reading, but even this has limits.
My understanding as well, as if Reddit was down and redditors took to HN to distill their anger. It's not calling a specific comment like this, it's calling most of comments here like this.
Perhaps about the quality of the comments. In my opinion it's not really that bad but reddit-style comments are often cheap shots that are slightly off topic, some of which I see present here
There are many differences IMO. Moderators, for instance, censor a ton on reddit; but the UI is also different. I much preferred old.reddit.com over both the new reddit (which is utter trash) and hackernews (which is simple, but also ... strange and awkward, much more confusing than old.reddit.com - and I hate the fact that after like 5 comments, I get locked out for hours before I can make more comments, that is the worst decision hackernews ever made).
I don't think length intrinsically denotes quality. One could say that longer sentences show more quality, but I would not even be certain of that either.
The biggest difference I noticed has been between smartphone/tablet users on the one hand, and oldschool desktop computer system users. I belong to the latter group and I think we, as a group, write more, and faster. I'd also like to assume it has a higher quality, but I am not automatically convinced of that either.
Always thought this rule is hilarious because the fact that such comparisons are so overwhelmingly common that they needed to address it with this bitchy little rule (adding one link to an example per word is a classic tell that an internet moderator is spending their highly compensated moderation time Extremely Mad lol).
However, rather than understand the obvious explanation for it, that reddit and HN's karma systems encourage performative comment behavior with people trying to be epic for the peanut gallery, they make the boneheaded decision to just assume it's an "illusion". It's like Dunning-Krueger but for their theory of mind and general emotional intelligence, with the cause being obvious enough that I don't feel I need to name it.
I suspect it'll only become more common as Reddit enshittifies further and further. The platform is basically unusable for anyone, thus its suitability as a 'containment zone' for the redditor phenotype goes down with it.
I hope BlueSky makes a Lemmy-style Reddit alternative, just something to satiate people's social media-ified forum fix.
Real talk I have a M-x unfuck-this-buffer. All it does is toggle the Unicode input mode that I sometimes accidentally trigger but have never figured out how.
Embarrassing that a multi-billion dollar corporation that other billion dollar corporations depend on seems to be fine with breaking their clients workflows. Was github always this shoddy or has something changed to cause this many outages?
> has something changed to cause this many outages?
AI coding. Which can be interpreted one of two ways:
* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or
* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.
That's a correlation but I've never seen a proof of causation. The variable can't be isolated because they also started shipping more features.
Also, the uptick in odd issues post acquisition pales in comparison to the scaling issues they've had over the past 1-2 years. As a matter of scale, trying to link this back to Azure doesn't really square. If anything they'd potentially be in a worse spot without having access to the resources of a massive public cloud..
They used to run their own data centers, and while the unicorn (their error page back in the day) was visible sometimes, it wasn't nearly as bad as things are today.
And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.
I can attest to the outage starting around 11:05, as that’s when my GitHub auth for Tailscale SSH broke. Of course, the next 10 minutes of troubleshooting and checking GitHub’s status page showed “all systems operational,” so I assumed it was something on my end. Then, it started working again.
And while I was typing this comment it just blipped again. Atleast this time I know why.
I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.
> I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.
Unless I've missed something, I don't think you can. I also had to use GitHub auth, as it was the least-bad option of the choices given. I wish they just did email/user + pass + TOTP like regular platforms.
You can't do just an email address, but if you're running your own domain you can do your own OIDC with it. My tailnet auths against my self-hosted KanIDM server.
1105 utc? I had (self hosted) runners running at 1336 gmt although when I had a quick look it felt a bit odd - jobs ahead finished but hadn’t updated the pr status
There aren't any restrictions anymore on who you can invite to your team. I added a passkey admin account and added other users via email invites. The account owner is still my GitHub account but I pretty much don't use it day to day.
GH reliability has been a regular question mark lately, and many time issues are not reported publicly too.
In the past 2 months I've seen 3+ instances where GH action were not triggered on commit pushes, leading to silent workflow failures, and seeing GH merge queue being stuck for 3+ hours for production product facing repos
I have to second this, GitLab CI is pretty good. I'll never be on board with infrastructure people's obsession with putting code in yaml but outside of that, the model makes much more sense to me.
A GitLab runner is a machine/VM which can get notified of a GitLab CI job, then run the job's container image (downloading it from GitLab's container registry if necessary), then run the code in the yaml file. You get to control your build environment.
It's so weird that GitHub Actions's model is "have one absolutely gigantic container image which contains everything any build could ever need, and if something's missing, the documented answer is to apt install it every run".
To be fair to GitHub, outages are pretty short (~40 minutes), and they rarely impact critical parts of the service, nor do they affect one’s ability to work (git offline)
Production has a critical error and we need to push a fix while our whole build pipeline runs on GH CI. Which is not working. Thank God their critical parts never break.
If I can't push my code, I'd say that's a critical issue. And if my Actions won't run (that I pay for), I'd also put that in the critical issue bucket.
I'm going to be investigating self hosting, this is ridicolous.
git push ending up with Internal Server error is similar to save file ending up with disk full error. yes, it can work after 40 minutes, and one can go to lunch and retry later, but here in Europe the troubles started at the end of a (normal) workday.
I'm not able to get code pushed, meaning I can't address PR feedback, meaning I can't get things approved, meaning code isn't getting merged to main and thus a ticket isn't getting closed. Sounds pretty critical to me.
One of the PRs I eventually managed to open 8 mins ago had a page flash up a generic error page containing a badge error of: "Unexpected token '<', "<!DOCTYPE "... is not valid JSON"
GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.
It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.
The drive to win a market by accumulating customer market share is no longer the main drive behind business ventures in our economy. It has been replaced with the drive to increase equities market valuation. This, at times, contradicts accumulating or retaining customer market share.
Not many people are interested in providing investment funding a meaningful competitor to GitHub. There are sexier, bigger bets out there than "providing a place for version control software to be hosted". The people who are frustrated enough with GitHub to self-host have the ability to do so. The rest of us just get up from our desks, go for a coffee break/walk, and wait it out, because migrating to self-hosted is a PITA and not a core business competency.
They'll keep having outages, and most will keep paying for it. If they're paying for it despite fewer resources being spent on uptime, that means the company's more valuable as an equities market participant. Line go up.
Premature declaration of Resolution is pretty bad user experience, if I’m being honest…
It's not getting better, if anything it's getting worse. Best time to get off GitHub was yesterday, the second-best day to get off is today. Tangled, Codeberg or self-hosted Forgejo (my approach) or Gitea are all good alternative solutions here.
They really should be partitioning their infrastructure so that their paying customers are not in the blast radius of outages from the legions of repos running their 700 test vibecoded regression suite every time they change a config file.
This misses the point in a bad way. You shouldn't have a single deploy funnel to begin with. You should be able to do deploys from multiple places.
Lots of things we should do, only a few we actually have time to fix. If I were choosing between "Migrating away from GitHub" and "Adding another way to deploy in case GitHub is down", I'd take advantage and do the first, because you'll end up having to replicate SCM, build infrastructure and so on anyway, why not do it properly?
[0] https://news.ycombinator.com/newsguidelines.html
It doesn't add to on-topic discussion.
This thread isn't "HN" and that comment wasn't saying it was turning into Reddit..
But let's call a spade a spade. I got worried my account would be flagged due to the amount of down voting I was doing.
The biggest difference I noticed has been between smartphone/tablet users on the one hand, and oldschool desktop computer system users. I belong to the latter group and I think we, as a group, write more, and faster. I'd also like to assume it has a higher quality, but I am not automatically convinced of that either.
> Why is GitHub down so often?
Short. Invites conversation. Shows intellectual curiosity.
> It's so over
Reactionary, performative, low effort slop that's out of place.
Unfortunately the latter is on the rise big time and overly prevelant in this thread.
However, rather than understand the obvious explanation for it, that reddit and HN's karma systems encourage performative comment behavior with people trying to be epic for the peanut gallery, they make the boneheaded decision to just assume it's an "illusion". It's like Dunning-Krueger but for their theory of mind and general emotional intelligence, with the cause being obvious enough that I don't feel I need to name it.
I hope BlueSky makes a Lemmy-style Reddit alternative, just something to satiate people's social media-ified forum fix.
C-x M-c M-unfuck-git
Did I break this? Making me insecure about changing anything on github.
1: The status site says that pull requests are working,
2: A PR I'm working on right now is missing commits,
3: The contact support page errors out.
Edit: 4 minutes later and I can push again
AI coding. Which can be interpreted one of two ways:
* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or
* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.
Also, the uptick in odd issues post acquisition pales in comparison to the scaling issues they've had over the past 1-2 years. As a matter of scale, trying to link this back to Azure doesn't really square. If anything they'd potentially be in a worse spot without having access to the resources of a massive public cloud..
The data speaks clearly: https://damrnelson.github.io/github-historical-uptime/
And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.
I wouldn't hold my breath here.
And while I was typing this comment it just blipped again. Atleast this time I know why.
I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.
Unless I've missed something, I don't think you can. I also had to use GitHub auth, as it was the least-bad option of the choices given. I wish they just did email/user + pass + TOTP like regular platforms.
https://tailscale.com/blog/passkeys
Maybe running your own OpenID service would work?
https://openid.net/developers/how-connect-works/
Too much work though. Easier to just switch to Nebula or Netbird and self-host the full stack.
There 4 updates contains webhook mention in 2 of the updates.
"Webhooks is experiencing degraded performance"
In the past 2 months I've seen 3+ instances where GH action were not triggered on commit pushes, leading to silent workflow failures, and seeing GH merge queue being stuck for 3+ hours for production product facing repos
A self-hosted enterprise GitLab instance in Azure, on the other hand, has had outages and performance degradation quite a few times.
A GitLab runner is a machine/VM which can get notified of a GitLab CI job, then run the job's container image (downloading it from GitLab's container registry if necessary), then run the code in the yaml file. You get to control your build environment.
It's so weird that GitHub Actions's model is "have one absolutely gigantic container image which contains everything any build could ever need, and if something's missing, the documented answer is to apt install it every run".
Maybe 30 minutes to get it all setup and then I moved on with my life.
I'm going to be investigating self hosting, this is ridicolous.
Same as with container registries; prod shouldn't be pulling from Dockerhub.
I noticed because SSO was down. They don't even report on that as far as I can tell.
A few hours ago GH pages just would not publish at all.. and the status page showed as fine, I guess it's now cascaded into a proper event.
GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.
It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.
Not many people are interested in providing investment funding a meaningful competitor to GitHub. There are sexier, bigger bets out there than "providing a place for version control software to be hosted". The people who are frustrated enough with GitHub to self-host have the ability to do so. The rest of us just get up from our desks, go for a coffee break/walk, and wait it out, because migrating to self-hosted is a PITA and not a core business competency.
They'll keep having outages, and most will keep paying for it. If they're paying for it despite fewer resources being spent on uptime, that means the company's more valuable as an equities market participant. Line go up.
https://github.com/about