On January 1, I received a bill from my web hosting provider for a bandwidth overage for $155. I’ve never had this happen before. For comparison, I pay about $400/year for the hosting service, and usually the limitation is disk space.
Turns out, on December 17, my bandwidth usage jumped dramatically - see the attached graph.
I run a few different sites, but tech support was able to help me narrow it down to one site. This is a hobbyist site, with a small phpBB forum, for a very specific model of motorhome that hasn’t been built in 25 years. This is NOT a high traffic site; we might get a new post once a week…when it’s busy. I run it on my own dime; there are no ads, no donation links, etc.
Tech support found that AI bots were crawling the site repeatedly. In particular, OpenAI’s bot was hitting it extremely hard.
Here’s an example: There are about 1,500 attachments to posts (mostly images), totaling about 1.5 GB on the disc. None of these are huge; a few are into the 3-4 megabyte range, probably larger than necessary, but not outrageously large either. The bot pulled 1.5 terabytes on just those pictures. It kept pulling the same pictures repeatedly and only stopped because I locked the site down. This is insane behavior.
I locked down the pictures so you had to be logged in to see them, but the attack continued. This morning I took the site offline to stop the deluge.
My provider recommended implementing Cloudflare, which initially irritated me, until I realized there was a free tier. Cloudflare can block bots, apparently. I’ll re-enable the site in a few days after the dust settles.
I contacted OpenAI, arguing with their bot on the site, demanding the bug that caused this be fixed. The bot suggested things like “robots.txt”, which I did, but…come on, the bot shouldn’t be doing that, and I shouldn’t be on the hook to fix their mistake. It’s clearly a bug. Eventually the bot gave up talking to me, and an apparent human emailed me with the same info. I replied, trying to tell them that their bot has a bug to cause this. I doubt they care, though.
I also asked for their billing address, so I can send them a bill for the $155 and my consulting fee time. I know it’s unlikely I’ll ever see a dime. Fortunately my provider said they’d waive the fee as a courtesy, as long as I addressed the issue, but if OpenAI does end up coming through, I’ll tell my provider not to waive it. OpenAI is responsible for this and should pay for it.
This incident reinforces all of my beliefs about AI: Use everyone else’s resources and take no responsibility for it.
I was like, damn $38/month! Then I looked at the company I use now, and their cheapest vps is $22. They’re a lot cheaper than I remembered.
But they also have less disc space than my shared hosting.
Huh now you have me looking. The price of the additional disc space I’d need is still less than I’m currently paying. I’d just about break even if I matched my current allocation of disc space, but I don’t use anywhere near that much. And it would do email, instead of being charged separately. Hmmmmmmmmmm!
(I was researching providers last year, and even $22 is still kind of expensive compared to others 😅
For data storage (not rootfs) I have their plan with €3.5/1TB/month; it’s a network drive that you can mount on the VPS with sshfs, webdav, samba etc, so you could check if your provider has something like that as well!
Oh for email I recommend mailu.io! It has a docker-compose generator, so the set up is pretty straightforward! I however have it on a separate VPS so it won’t be affected when I’m trying to deploy other services etc (e.g. the configuration of the reverse proxy and certificates), and with webmail and spam filtering etc it’s quite a few containers anyway 😅 so I’d like them to have dedicated resources and not be affected by the resources used by the other services
You’re right, I’m doing some looking and …definitely some less expensive options out there.
I’m always a little nervous because I switched to another company years ago, and it was a mess - frequent down time, missed connections, etc. I switched back to my current host within a year.
I could also switch to AWS. I already have an account (I use their S3 glacier service for backups). Advantage would be that I’d pay for exactly what I use…
It doesn’t have to be FAANG for it to be a good service 😅 (you unfortunately probably just found an outlier 🥲) In fact, AWS is one of the most expensive, and when I looked into them (I think it was AWS) the server options were way too complicated 😅 And I couldn’t find advantages to most use cases compared to others to justify paying so much more
Hetzner I think is one of the largest providers in Europe, Scaleway is very good as well (although still a bit expensive but with more flexibility but you may not always need it), I remember some friends and our local hackerspace’s server used Leaseweb a while ago and didn’t have any complaints, and I’ve heard good words about Ionos as well.
For storage there’s Backblaze, Wasabi, DigitalOcean (and again Scaleway and Hetzner); also TIL https://www.s3compare.io/
There are so many good options out there 😅
You can still easily spin/scale up/down instances on most providers: Hetzner and Scaleway have APIs and CLI utilities, and they charge by the hour. But even more importantly, even if you use fewer resources, for me I found that e.g. having a VM off 50% of the time on AWS was still much more expensive than 100% on Hetzer 😅
Yeah, I was mostly kidding about AWS. It really is complicated, and there are bandwidth fees and storage fees and this fee and that fee… I also don’t want to be worrying about how much each month is costing me like that. I’d rather pay a fixed rate and not worry about it.
I’m a little hesitant on a European provider only because practically all of my users are in North America. I know it would work fine, but it makes sense to me to locate where I am and where the users are.
My contract with this company ends in July, so I’ve made a note in my calendar for June to find a VPS host. Thanks for the insight, you definitely have me thinking. I had no idea I could get a VPS less expensively than I’m currently paying…