The Gerudo Fortress is on the other side of Gerudo Valley, and entrance to the Haunted Wasteland. If seen by one of the Gerudo they will not waste a second to apprehend you and kill you, or just throw you in their jail and several day later kill you. But the Gerudo have been known to give membership passes to move freely around the fortress to certain individuals that prove themselves worthy.