The whispering that pulls me away seems to come strongest exactly when I begin to pray. Why does my mind attack me only in salah, and how do I answer it?

The whisper grows loudest at the door of the prayer, and there is a reason for that. It is not a sign that Allah has left you.

A thief does not trouble himself over an empty house. He waits near the door of the richest one. When you stand before your Lord you are, in that moment, the wealthiest of people, so that is where the pressure comes. The fierceness of the distraction tells you something about the worth of what you are doing, not about the state of your heart.

This trouble is as old as worship itself. A companion complained of exactly this, and the Prophet (peace be upon him) named it and gave the remedy.

“That is a shaytan called Khinzab. When you sense him, seek refuge in Allah from him and spit lightly to your left three times.”

The Prophet (peace be upon him), Sahih Muslim (2203a) · meaning

Allah also told us plainly who this whisperer is, so that we are never fighting something unnamed.

“…who whispers into the breasts of mankind, from among the jinn and mankind.”

Qur’an, An-Nas 114:5-6 · meaning

So do not argue with the whisper, and do not try to trace where it came from. That is a maze with no exit. Turn away from it the way you turn away from a rude noise in the street, and take refuge in the One who is stronger than every whisper.

“And if an evil whisper from Shaytan tempts you, then seek refuge in Allah.”

Qur’an, Al-A’raf 7:200 · meaning

Your effort to come back to the prayer, again and again, even before any sweetness has arrived, is counted with Allah. The struggle is not a mark against you. It is your prayer being fought over, because it is worth fighting for.

One small step: Before you say “Allahu Akbar” to begin your next prayer, pause for one breath and say quietly, “A’udhu billahi min ash-shaytan ir-rajim,” seeking refuge before the whisperer even reaches the door.