I’m looking at starting a service that involves hosting a lot of LLM models, which are often going to be 16GB+ (compressed). I did a bit of searching for cloud storage providers with cheap egress, and the cheapest I could find is $0.01 per GB, which would still be $0.16+ per download.

How do sites like Huggingface or CivitAI do it? Lots of VC funding?

  • Oliver LoweEnglish
    8•1 year ago

    If the files are not going to be changing much, then what is typically done is to use a CDN service (e.g. Cloudflare, Akamai, Fastly). The idea is you have an “origin” which could be any old server which serves your files over HTTP (even a VPS running nginx). The CDN is configured to proxy requests to the origin, building up a cache of the files it serves. The CDN can serve files from cache on their own (very large) infrastructure. See also What is a CDN?

    • Oliver LoweEnglish
      6•1 year ago

      So I got curious and wondered how HuggingFace hosts their files. It’s AWS CloudFront:

      $ curl -LI https://huggingface.co/facebook/musicgen-large/resolve/main/state_dict.bin

HTTP/2 302 
content-type: text/plain; charset=utf-8
content-length: 1198
location: https://cdn-lfs.huggingface.co/repos/81/...
date: Fri, 23 Jun 2023 07:09:39 GMT
x-powered-by: huggingface-moon
x-request-id: Root=1-64954533-1eb79eed4ea500203f6435cb
access-control-allow-origin: https://huggingface.co
vary: Origin, Accept
access-control-expose-headers: X-Repo-Commit,X-Request-Id,X-Error-Code,X-Error-Message,ETag,Link,Accept-Ranges,Content-Range
x-repo-commit: c19300a6b2b62d29b345ae9eb7b163278e65238a
accept-ranges: bytes
x-linked-size: 6514108999
x-linked-etag: "1f0cf17b5e65c5dd8ba71767371c377f174e4ce1db44bc4d6657825769f26ffd"
x-cache: Miss from cloudfront
via: 1.1 65c7d0c3355767ac8658c2122c8280b6.cloudfront.net (CloudFront)
x-amz-cf-pop: SYD1-C1
x-amz-cf-id: XAU8X4yneUeudylCi_9MeAYZmISCr8OHiBcgjAGcBQT-edrBF6wGCA==

HTTP/2 200 
content-type: application/octet-stream
content-length: 6514108999
date: Fri, 23 Jun 2023 07:06:20 GMT
last-modified: Thu, 08 Jun 2023 19:05:02 GMT
etag: "44ef1b51c0cc2200e29fed5cddbf8e27-408"
x-amz-storage-class: INTELLIGENT_TIERING
x-amz-server-side-encryption: AES256
x-amz-version-id: upVN9_QvGmQZMDWfVECqbytHgWjHTz4t
content-disposition: attachment; filename*=UTF-8''state_dict.bin; filename="state_dict.bin";
accept-ranges: bytes
server: AmazonS3
x-cache: Hit from cloudfront
via: 1.1 dff3fc94ddb54b32b708edf2668b23d2.cloudfront.net (CloudFront)
x-amz-cf-pop: SIN52-P1
x-amz-cf-id: JWTqKUDp8bVPD7Tt4DBySj8zbgT8G60sCA5BMC1cYF4vK9nx45f1Iw==
age: 200
vary: Origin
    • @minorninthEnglish
      1•1 year ago

      That’s true, but I just checked a few CDNs and you won’t find one for less than $.01/GB. The lowest I found was $.03/GB.

      • Oliver LoweEnglish
        4•1 year ago

        To keep costs down and depending on how much you want to get your hands dirty, you could start investigating renting dedicated servers. Some hosting providers offer unmetered network connectivity. Here’s something from OVH: https://www.ovhcloud.com/en/bare-metal/rise/rise-stor-1/

        And hey, depending on how grassroots the project is, there’s always bittorrent! ;)

        • @[email protected]OPEnglish
          3•1 year ago

          I was considering this. The hosting provider we use for model training runs doesn’t charge for ingress/egress. Their storage costs would eat us alive though haha. OVH looks much more promising.