Hey folks!

For the past few hours, lemm.ee has been bombarded with abnormal (almost definitely automated) traffic from a range of different IP addresses. This managed to overwhelm our servers, and we were offline for the past hour or so.

I was in the middle of celebrating my birthday, so response was a bit slow, but I believe we are recovering now, with mitigations in place to try and prevent further issues. Some of you may be inconvenienced by some bot checks when you browse lemm.ee, I am sorry about that, but it's necessary for now.

Sorry for the issues and I hope you have a nice weekend ahead!

Hey folks

Just a quick heads up, we will be performing some database maintenance today. Expected downtime is ~15 minutes.

Sorry for the inconvenience!

---

Update: maintenance complete!

Hey Folks

Just a quick note to let you all know about some changes in the lemm.ee admin team. After discussing things with the other admins, we've decided to shuffle around our roles a bit.

Up until now, I’ve been the head admin at lemm.ee - handling infrastructure, maintaining rules and policies, and acting as the main contact person for the admin team.

However, I’ve come to realize that this role has taken a toll on me. While I still love the idea of Lemmy and everything it stands for, being an admin has slowly drained the joy I once had for the platform. The occasional negative experiences have been increasingly difficult for me to shake off. For the past several months, I’ve found myself hesitating to check my DMs or the moderation queue, simply because I’m bracing for some new drama that I no longer have the energy to manage.

After some conversations with the team, we’ve agreed on a plan to ensure my burnout doesn’t negatively impact the instance:

1. I am stepping down as head admin of lemm.ee.
2. The new main contact person for the admin team will be @[email protected].
3. I’ll continue to maintain and update the infrastructure behind the scenes.
4. The rest of the admin team will now handle all moderation issues, managing our policies, and any general admin communications.

It’s been an honor to serve as your head admin, and I’m incredibly grateful for the amazing people I’ve met here. I’m excited to stay involved in a capacity that works better for me and allows me to enjoy this community once again.

See you around!

Hey folks!

I'll be updating lemm.ee to 0.19.7 shortly. I'll take it offline & also perform some database maintenance at the same time.

I have been investigating some incoming federation issues, and I haven't been able to find the actual cause. My current theory is that they are related to some server timeouts. I am hoping that the combination of the Lemmy upgrade and database maintenance may help improve the situation, but if not, then I will try other approaches.

Edit: the upgrade has been completed!

Hey!

Unfortunately, Hetzner (our hosting provider) is currently experiencing some network issues. They are planning to address this with an emergency maintenance in roughly 13 hours from now, which will cause lemm.ee downtime. Hopefully we'll be fully recovered later tomorrow!

---

UPDATE: Sorry for the false alarm, I was on the move when I posted this and missed the fact that the Hetzner notice was actually for next month! So it's not as imminent as I originally understood. As we have a whole month to prepare, I will probably be able to come up with some alternative solution to prevent the downtime while they are conducting this maintenance.

Hey folks!

I am looking for feedback from active lemm.ee users on what you all value when it comes to images on Lemmy. I'll go into a bit of detail about what our options are, and then I would ask you to voice your opinion about the issue in the comments.

First, some context for those who don't know. Lemmy software can be configured to handle images in three different ways:

1. Store images locally - whenever an external image is posted somewhere, lemm.ee will download a permanent local copy. When you view posts, you are seeing our local copy of the image.
2. Proxy all images - similarly to the first option, lemm.ee will download a local copy of external images, however, this copy is temporary. It will be automatically deleted shortly after, and if users open the relevant post/comment again in the future, there will be another attempt to download a temporary copy at that point.
3. Pass through external images directly - lemm.ee never downloads any external images, users will always connect directly to the source servers to load the images.

There are pros and cons to each configuration.

Storing images locally

Benefits:

1. Your IP address is never leaked to external image hosts, as you never connect directly to the source server. External image hosts only see the IP address of the lemm.ee server.
2. External servers don't become bottlenecks for opening lemm.ee posts. If an external server is slow, it won't matter, because the image is always available locally

Downsides:

1. As time goes on, our storage will fill up with hundreds of gigabytes of useless images, most of which will never be viewed again after the relevant posts fall off the front page.
2. Many big external image hosts will rate limit bigger Lemmy servers, causing broken images when we fail to make a local copy.
3. Crucially: some people love to spend their time uploading illegal content to online servers. There are tools to try and filter out such content, but these are not perfect. The end result is that there is a high chance of some content like this inadvertently reaching lemm.ee storage and staying there permanently. This downside is why lemm.ee has not, and will not, use this particular configuration.

Proxying images

Benefits: In addition to the same benefits as exist for the permanent local storage, by only temporarily making local copies for the moment they are requested by our users, we free up a ton of storage & remove the risk of permanently storing illegal content on our servers.

Downsides: The key downside is that external rate limits hit us much harder, as we will be requesting external images far more often. This results in a lot of constant broken images on lemm.ee.

Passing through external images

Benefits:

1. Images are rarely broken, unless the source server goes down.
2. The images never touch our servers, removing a lot of risk with illegal content as well as with storage costs.

Downsides:

1. Our users lose a degree of privacy. Every external image that is loaded on your browser will result in the remote server getting a request directly from your computer to fetch that image - this is pretty much the same as you had visited that external server directly, which lets them log your IP address if they wish.
2. When remote servers are slow, it can slow down the entire page load in some cases.

Current situation

Initially, lemm.ee was using the third option of passing through images. Ever since support for option 2, image proxying, was implemented in Lemmy code, we immediately switched to that option, mainly for the privacy benefits. However, after many months, and being blocked by more and more external servers, it is clear that image proxying is seriously degrading the user experience on lemm.ee. We often end up with broken images, and our users have to deal with the results.

I still believe image proxying is a really valuable feature, but I am starting to believe it is a better fit for small instances which make much less requests to external servers.

As a result, I am now seriously considering switching back to the previous method of passing through external images.

This is where you come in - I would ask you as users to please let me know which do you value more: the privacy that you get from image proxying, or the better user experience you get from directly passing through images from their source. Please let me know in the comments how you feel. If I get enough feedback about people being against image proxying, then I will be switching it off for lemm.ee soon. Thanks for reading & sharing your thoughs, and I hope you have a great weekend!

Hey folks!

Unfortunately, roughly 2 hours ago, lemm.ee went offline. The cause was our load balancer: it suddenly decided that all of our servers had become unhealthy, despite all health checks responding successfully when I requested them directly. In such cases, the load balancer stops serving all requests, effectively meaning that lemm.ee is unreachable for all users. I am still not sure what exactly caused the issue, but I will try to investigate more over the weekend.

For now, we have partially recovered, and I am continuing to work on remaining issues. Hopefully we will be back to 100% very soon. Sorry for the inconvenience!

Hey folks

Just a heads up that I will be doing some minor database maintenance shortly. I expect the downtime to last <5 minutes.

Have a nice day!

Update: maintenance is complete!

Hey, folks!

Today, we can celebrate the first anniversary of the creation of lemm.ee! I thought it would be cool to write down how lemm.ee was born, as well as collect some stats about our first year. Here goes!

A quick recap of the beginning of this instance

As probably many others here, I discovered Lemmy early last summer. I had been aware of the Fediverse previously, and always thought it was an amazing concept, but I had never been super interested in Twitter-style social networks. When I found out that Lemmy combined all the great parts of federation with the best parts of link aggregation, I knew that I had to join immediately.

As I was trying to find an instance to make my account on, I realized that most instances were struggling to keep up with a massive influx of new users. At the time, there was a big explosion in Lemmy user numbers, and the network wasn’t fully ready for it. I have some experience with building software for scale, so it felt natural to set up a new instance and try to help with spreading out the load. I got to work in the evening of the 8th of June, 2023, and I was actually so excited about everything, that I completely skipped sleep that night. By the morning of the 9th of June, lemm.ee was online.

From the very beginning, I always intended for lemm.ee to be a welcoming, reliable, and stable gateway into the Lemmy network. I wrote a welcome post on lemm.ee, which most of you have probably seen, as well as a comment on lemmy.ml inviting new users to lemm.ee (lemmy.ml, as many instances, was extremely overloaded at the time).

We started growing extremely quickly. Thousands of users joined lemm.ee over the first few months. Even during the biggest waves of new users, we never closed our sign-ups. The first month or two were definitely very stressful in terms of just trying to deal with the load, but overall, I think I managed to deal with it well enough, and lemm.ee has been running more or less smoothly (with a few exceptions) ever since.

Some stats about the first year

I promised to collect some statistics about lemm.ee so far. This is what I’ve come up with:

Usage

Overall, lemm.ee has 28,715 registered users. Of course, it’s easy to create an account, and most of these are probably inactive at this point, but it’s still a ridiculous amount.

Of all the registered users, 7903 have made at least one post or comment.

7373 users have never made any posts or comments, but have still been voting. This means that out of users who actually interact on lemm.ee, more than half generate content (through their comments and posts) - this is way more than I expected!

Meanwhile, we also have 13,439 users who have never made a single comment, post or vote. I guess most of these are people who just signed up and never got into Lemmy, but I’m sure there are quite a few hardcore lurkers among this group as well.

As for communities, our users have created 1430 of them. Most of these have not (yet) taken off, as only 491 of these communities have at least one comment in them. In general I am happy to see some great communities appearing on lemm.ee - my hope is that we can spread awesome communities out quite evenly on the network, so that in the end, no instance becomes a single point of failure for Lemmy.

Judging by posts and comments made by lemm.ee users, I feel like we’re definitely on the right track: our users have made 20,898 posts in local communities, and 30,847 posts in communities hosted on other instances. The situation is even better for comments, where lemm.ee users have written 42,785 comments in local communities, and a whopping 569,730 comments on remote communities! This means that lemm.ee is not just its own little closed pocket in the Fediverse, but indeed a proper gateway to the Lemmy network, which is exactly what I always hoped it would be.

Note about comment and post counts: I realize the numbers above don’t match the stats about posts and comments on our front page, I’m guessing something is out of sync there, but the stats I am sharing here are based on actual fresh data, counted directly in our database today.

Lemmy (and lemm.ee) would be quite useless without its users, so a big thanks to all of you for using lemm.ee!

Administration

We have a really awesome volunteer admin team, with admins putting in countless hours of their free time to help weed out bad actors. A lot of the work our admins do is completely invisible to most users. I think the admin team does not really get enough recognition, and in fact in many cases, they actually get some undeserved abuse thrown at them.

I am personally very grateful for everybody who has stepped up to be a part of the team, and I think all lemm.ee users benefit from their work every day. In the past year, our admins have handled 12,329 reports from users. While most reports aren’t too bad, and don’t require harsh action, there is still a significant amount of these reports which contain the absolute worst content which you can find on Lemmy - hate speech, bigotry, gore, even illegal content. Our admins are constantly going through every single report they receive, to ensure that mods are getting admin-level support where needed, and to ensure that malicious users in general can’t use lemm.ee to spread garbage into the Lemmy network.

In terms of admin actions, I think the most interesting statistic might be amount of users banned by lemm.ee admins, grouped by their home instance. I will list the top 10 instances here:

• kbin.social: 581
• lemm.ee: 355
• lemmy.world: 31
• sh.itjust.works: 29
• m.mxin.moe: 28
• discuss.tchncs.de: 26
• kbin.chat: 22
• mastodon.social: 19
• lemmy.ca: 18
• fedia.io: 16

As you can see, with the exception of kbin.social, the vast majority of our instance bans are for our own users. Most of the big instances are actually very good at banning their own abusive users, and once they are banned on their own instance, our admins don’t really need to worry about them, as they have no way to log in at that point. kbin.social is a bit of a special case - they either don’t give out a lot of bans, or those bans just don’t federate to Lemmy properly, and for some reason, a lot of advertisers sign up on that instance all the time.

Financials

I have received some questions every now and then about how much it costs to run lemm.ee. While you can always get a sense for the predicted monthly costs for the current month on https://status.lemm.ee, I thought I might include a full breakdown of our costs for the first year here.

Here are all of our costs for the past year, grouped by service:

• Postmark: 177.06€
• Cloudflare: 222.28€
• DigitalOcean: 1744.27€
• Hetzner: 510.20€ (lemm.ee migrated from DigitalOcean to Hetzner several months ago)
• Backblaze: 3.78€ (we’ve been using Backblaze B2 for a few months now, it’s incredibly cheap)
• Domain registration: 100.70€ (paid for the next 10 years!)

We are currently completely funded by lemm.ee users!

There is a small minority of users who are shouldering the entire cost of lemm.ee for all of us. I am extremely grateful that others find Lemmy useful enough that they have put their own money into ensuring financial stability for lemm.ee.

We currently have 49 active sponsors on GitHub, and 7 active supporters on Ko-Fi. In addition, there have been 62 more sponsors on GitHub over the past year, as well as 49 additional supporters on Ko-Fi. This means that in total, 167 users have supported lemm.ee financially. This has completely exceeded all my expectations, I really think it’s incredible. A huge thanks on behalf of myself (and I think I can speak for all other lemm.ee users here as well) to all the supporters!

Conclusion

Running lemm.ee has certainly been a rollercoaster in many ways. There are a lot more things which happened during the first year that I could write about here. On the other hand, this post is already quite long, and a lot of the things which happened are probably best forgotten about anyway, so I think I should wrap up here 😅.

At its core, Lemmy is really an amazing piece of software. It’s helping real humans connect on the internet, without any corporate bullshit. I am very happy to be here with all of you, thank you for joining lemm.ee for its first year, and I hope you’ll join me here again when I write this post in another year from now!