It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has focused on training increasingly large and powerful self-supervised (language) models. Since these large language models exhibit impressive predictive capabilities, they are well-positioned to be strong compressors. In this work, we advocate for viewing the prediction problem through the lens of compression and evaluate the compression capabilities of large (foundation) models. We show that large language models are powerful general-purpose predictors and that the compression viewpoint provides novel insights into scaling laws, tokenization, and in-context learning. For example, Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to 43.4% and LibriSpeech samples to 16.4% of their raw size, beating domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively. Finally, we show that the prediction-compression equivalence allows us to use any compressor (like gzip) to build a conditional generative model.

I wonder what a paper like this, especially given the title, does for the legal case regarding copyright and generative AI. Haven’t had a chance to read the paper yet, so don’t know if the findings are relevant to copyright.

  • wagesj45
    19 months ago

    In a majority of the cases this simply isn’t true.

    The neural networks did not spring from the ether. And they are not naive neuron grids so simple as to be trivial. There are multiple layers with multiple purposes that have different designed functions.

    Yeah, there’s some people deep into the ML game, but most predictive engines aren’t using any kind of additional fine tuning or dataset from their users.

    And there are relatively few people who design the image sensors for cameras compared the the number of people using a camera to take pictures. They’re still designed as a tool by a person.

    trained on many works that were privvy to copyright

    Trained the same way you learn with the wetwear neural network in your brain. And even if you’re not convinced that these networks “learn” the same way we do, the resulting network weights are entirely transformational, which is perfectly allowed by copyright law. With 5 billion image/text pairs for training into 960 million parameters in the diffusion and text encoding networks of stable diffusion, for example, that is 0.2 parameters (or about 6 bits), per image in the resulting product. The image, as such, is almost entirely discarded.

    And morally, what’s law now IS what is right.

    I fundamentally disagree with you and I do not think we’ll come to an agreement on this. There is a lot I find morally and philosophically wrong with our copyright law, and the current findings of the courts regarding AI works is just a fraction of that.

    Typing some sentences into an image diffusion algorithm is no different than plugging an equation into a calculator.

    A camera’s image sensor is just as deterministic as the neural network weights. The human work comes from the judgement used when conjuring a prompt to feed into the tool, just like a photographer decides what light reflecting source to point his camera at.

    There are clear lines drawn in the sand that most people agree with morally.

    I’m not convinced the lines are either clear or agreed upon by the majority. This is a really complex set of circumstances and there’s a reason we’re still battling it out in the courts and in online forum comment sections. ;)

    • @KingRandomGuy
      28 months ago

      And there are relatively few people who design the image sensors for cameras compared the the number of people using a camera to take pictures. They’re still designed as a tool by a person.

      I’m not the most familiar with copyright law, but IIRC you’re certainly able to violate copyright while taking a photo. If you take a photo of a copyrighted work (i.e. parts of a book or something) without artistic intent, I don’t believe that’s considered transformative.

      I suspect the courts will end up having to deal with many of these issues on a case-by-case basis, just like they already do with fair use.