To clear or remove an added chart item, move the mouse pointer over to the chart item, left-click on the item and press the Delete on keyboard.
Alternatively, you can use the Eraser functions. Click on the Eraser tool and then click on the chart item to be erased. You can also use the Erase All tool to remove all manually added chart items.