The speed at which you are connected to the internet via your ISP is only part of the path to youtube’s content servers. You could easily have a fast connection to an ISP with a congested uplink, you could be located a long way from the nearest google CDN, or that CDN could be overloaded.