Are you talking about wlr-randr? Because its --transform argument only accepts normal|90|180|270|flipped|flipped-90|flipped-180|flipped-270, not any transformation matrix. Maybe its just a limitation of the command line tool and it could be modified, I don’t know, I haven’t been too deep in the code or the protocol specifications. It also looks like it only works with wlroots based compositor.