Ancient Jews relied on a passage from the book of Exodus (34:6-7) to conclude that man's capacity to sin can never exceed God's capacity to forgive. Everything could be forgiven, provided that the sinner contritely confessed and resolved to avoid that mistake in the future.
In the Ashkenazi folk ceremony of tashlikh on Yom Kippur, Jews go to water, preferably a river or a sea full of fish, and shake their clothes as if to cast off every trace of sin, while reciting appropriate verses, such as Micah 7:18-20, which contains the words "and you shall cast [tashlikh] into the depths of the sea all their sins." This one example (among others, such as the scapegoat) supports the idea that the slate could be wiped clean: a sinner could start all over again. Certainly, the Roman Catholic theology of confession also involves centrally the belief that the slate can be wiped clean: After proper atonement, a Catholic's sins are washed away.
Joe Biden is a practicing Catholic, which is to say that Biden comes from a faith community enthusiastically committed to the idea that sins really can be forgiven. Calvinists present a counterexample here; unlike most Jews and Catholics, Calvinists do not believe that all sins can be washed away. Calvinists (and various other Protestant communions) also reject the Catholic distinction between "mortal" (very serious) and "venial" (less serious) sins. Although subsequent Catholic theologians have disagreed with him, it is worth noting that Thomas Aquinas, a lynchpin of the Catholic moral tradition, concluded that only a mortal sin (such as murder or adultery) deserves the name of "sin" (Summa Theologica I-II, q. 88, a. 1).