Bus drivers in Brighton and Hove, on the south-east coast of England, have been told by their bosses not to call female passengers "babe". There has apparently been just one complaint that the term is sexist. Other regional organisations have banned, or advised against, other terms of endearment, too - Newcastle City Council asked its staff to think very carefully before calling women 'pet' or 'hinny' (which comes from 'honey').
Love, duck, dear, chuck, lover, guv, hen - these are mostly traditional and long-standing regional terms of endearment and are used to address people of all ages. I wouldn't necessarily say that "babe" belongs in this group though; it is probably a more modern expression. As Ian Brookes from Collins says in this BBC article, it became popular in the 1990s to describe an attractive woman, thus it is seen as having overtones of sexism.
Words out of context are fairly neutral. Most people don't mind a shop assistant or market stall holder calling them 'love' or 'ducks' in a cheery voice, but would probably object if a boss said it. A young woman might not mind an elderly man calling her 'pet', but an older woman probably wouldn't like to be addressed in this way by a young man.