Why not just use standard thermostat functionality: set the target temp a bit higher when rates are low and a bit lower when rates are high.
That was my original idea and it actually works pretty well, but since the cost of power spends most of the day at industry average rates electric heating gets pretty expensive which is really what I'm trying to minimise.
One thing you don’t mention is whether you have any way to store heat
I don't, but I really, really wish I did. The place I'm in is rented so I'm loathe to make big changes like installing storage heaters (installing relays in the walls behind the current radiators doesn't count, shush) but I had old-fashioned, 1980s storage heaters exactly as you described in my old place and I loved them for the exact reasons you described. They weren't active with a fan, but even just having a very heavy, very hot thing in the corner of the room was enough to maintain the temperature and given my electric rates regularly get below 5p/kWh and sometimes even go negative overnight my heating bill was basically negligible. Consider me a member of Team Storage Heaters.
As you suggested, what I'm trying to do is turn my walls, floors and furniture into the thermal mass of a storage heater, by making them toasty when it's cheap in the hope they'll keep the room slightly warmer when it's expensive.