Cultural issues- certainly the hand gesture meaning would impact if thumbs up/down "works".
As for judgement- the second you add scoring- you are judging (ranking, rating). So, it's out there.
Question- if someone answers a question incorrect on an assessment, how is this indicated (common answers are "red X, green check", "Incorrect"- sometimes in red, etc...). The fact that you add an icon (thumbs down) doesn't seem much more harsh (the second you say "wrong"- or even "0 points" it's out there).
For me, the key thing is clarity. Is it clear whether the person "met expectations" or not? I've seen some companies try to be so sensitive about feedback, they diluted the impact of the message so much ("not the best answer") that the learners actually thought the answer provided may have been viable (when it was not).
As a counterpoint- sometimes you don't need to clearly say "correct/incorrect", "pass/fail", but just show consequence of action (i.e. show actual business results of "failure"- you didn't make the sale, your computer gets infected). A clear outcome that maps to real-life consequence may resonate more than just being told correct/incorrect.