^ agreed.
but to be fair for Issei, "skinship" within the Gremory has somewhat been desensitised and separated from the word "love". Hell, Issei's first interaction with Rias was him being naked with her in bed
Rias kinda "dug her own grave" by proclaiming the Gremory were very affectionate, even with their servant, thus Issei just made the assumption that there was no other meaning behind Rias' action.
Of course, Issei being Issei, he never questioned or noticed that only he himself had been receiving Rias' affection e.g. Rias has never kissed Kiba or Gasper the way she did Issei.
OR, Issei in the moment just had the biggest brain fart of his life when Rias asked him the question