There are many factors in determining the amount of energy used by any one person in any one elevator ride. Factors like the number of people in the elevator, the elevator’s capacity, whether it is going up or down, and the age/type of elevator.
One estimate says the amount of energy it takes for one person to travel one floor in an elevator is 2.5 Wh, roughly equivalent to charging your cell phone battery a few percentage points. Multiply this by all the people taking all the trips in elevators every day and you have a significant amount of energy being used for this intra-building travel.
Choosing to use your legs saves electricity and is also healthy for you. You’ll burn around 5 calories per flight of stairs.