The Keys of Heaven are a symbol of authority drawn from Isaiah.

Papal infallibility developed from a number of observations, such as the keys, Peter as the rock, and generally just the authority of the Pope.