Sounds like a loose power connector on the keyboard. Casio wires up really cheap power jacks on their keyboards with hair thin wires held only by a dab of solder.
My advice is to get it fixed before a short or a power glitch fries your board.
If it's outside of warranty then I'd advise bypassing the power jack entirely, drilling a hole in the back of the case and wiring in a better quality power jack.
If it's still under warranty then I'd advise sending it to Casio for repair.
In the meantime if you still have to make use of the keyboard disconnect the wall adapter entirely and run it on batteries.
That will also help isolate the fault to the power jack.
Gary