Disable the keystrokes of the datepicker unless focussed

Not sure if this is possible, but currently the date/time picker always pops up when editing hre start or end date of an event. This picker then catches cursor-left and cursor-right keystrokes, causing them to switch the event one day in the past or future.

If one just wants to edit e.g. the time and uses the cursor keys, this causes the date to be changed by one day and resets the time to 00:00. The latter is quite annoying, as the original time is lost and cannot be restored (no undo-key, as far as I know).