A bit more fantastic idea: Build it into a cliff in a mountain that looks like Devils Tower (the cave need to be unapproachable from above) tall enough that one would need a very long ladder, or a master climber, to reach up. Guarded on the bottom of course, to prevent anyone from trying. And a locked door, so if anyone got up they would still need to pick the lock while hanging on for dear life.

If you are rich and powerful, you have this machine. A medieval scissor lift basically, made by the best smiths in the kingdom. Need about 1K people to move it from the castle to the cliff. Need about a 100 people to walk in wheels to get the thing to elevate with a single but very important guy on the lift. He goes up, unlocks the door with his key, steps in, does his business, flushes, goes out again and locks the door after him, ring a bell, and descends.