The last two or three months, the site has been the target of unusual activity with a huge amount of requests.
It doesn’t impact the forum stability, but vastly increases the CDN [1] cost, from ~1$ a month to 50$ in August (so far) .
So, first, I’d like to thank you all guys again for the past donations. As said in the topic,
We’re exactly in this case. No need to donate right now, that’s not necessary. There have been enough donations so far, so I don’t have to pay anything with my own money .
I initially blocked the three countries the requests were originally sent from ( Singapore, Hong Kong, and Mexico), but I think that would only work for a time.
At first, the requests only came from Singapore, but this is not the case anymore.
Blocking a country doesn’t prevent users from using the site, they just wouldn’t benefit from the loading speed given by the CDN. It kinda defeats the purpose…
So I’m configuring stuff in the CDN administration to help mitigate the problem without blocking any countries, and hope unusual request amounts will be automatically detected and blocked.
If any of you encounter any issues while using the site, just let me know and I’ll have a look at it
A service that fastens the content for users all around the globe, useful since this is an international forum ↩︎
This issue is unrelated (expected issues would be more like some trouble to access the site – which would make seeing my topic or contacting me difficult ), but what does it show if you click on ?
Here you can block all servers using the AS number.
Many are attacking with hackers or attempting form spam.
My website is bombarded daily by all kinds of malicious programs. Many come from Hetzner servers and Huawei Cloud.
I didn’t really realise the scale of this AI bot/scraping issue and the percentage of global traffic it represents. The Register has various articles on this, I just read this one yesterday. According to one of the comments this traffic originates from lots of companies worldwide, so it it is not just your Meta/Google/OpenAI/Anthropic etc doing it, so it seems plausible that this may be the cause.
Is it worth finding out how much one of these bot-blocking services would cost – or do you think that would just be ridiculously expensive?
Some crawlers declare themselves as crawlers, but many don’t, and are harder to detect.
AI seems to have huge adverse effects, both in Internet pollution and resource usage (which also causes environmental issues).
Just to show how big the impact is for unicyclist.com, here’s one-month traffic (day 1 to day 28) in April, then in August (look at the Y-axis units!) :
Besides the huge activity spikes (like 55 GB in one day) these days, even a regular day with no spike is at least three times a regular day before the spam.
And it results in forty times the regular cost, which is absolutely not sustainable.
If I’m not able to fix the issue, I’ll block Singapore, Hong-Kong and Mexigo again from using the CDN, but that won’t completely solve the issue, and it might come back from other locations.
That is just insane, as you say, that level of traffic is unsustainable from a cost perspective. These people may have to pay for their datacentre but they expect others to pay for the bandwidth to feed it content.
This is just a thought, which may be completely impractical, but given the historical usage of this forum and reasonably light number of new posts, I would suspect most of the spam traffic is for really old posts especially compared to legitimate requests; would it be possible to restrict access somehow to posts by date? Having to be logged in to access a post more than (say) 2 years old would be a bit draconian but might stop these things sucking up all the historical content.
Anyhow, you know a whole lot more about this than me. I was really quite surprised the effect these things are having on internet traffic.
My work is getting hit by some bot as well which is causing all sorts of issues as it’s clogging up a booking system by putting in 10k requests in within 30 mins every 3 hours. Every time it gets blocked, it changes it’s country of origin and it’s script slightly.
All this wasted energy and additional cost because some tech bro has a get rich quick scheme…
No, it’s not possible. I think it would also vastly harm SEO, prevent visitors from seeing many interesting information.
The only thing possible is putting topics in specific additional categories only viewable by registered members, it has been done for certain old topics or categories that are not relevant anymore.
Honestly, I don’t know very much. I’m just helping the unicycling community stay active online and keep all the information available, but I have no serious technical background.
.
.
Singapore, 
Hong Kong, 
and Mexico), but I think that would only work for a time.
), but what does it show if you click on 
?
f-pulse:
)
Thanks for all you do.