Jev 缺乏内置概率校准功能，需额外标注数据进行校准

原话

I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%. I also tested with a fair coin 200 times and got 0.92 probability. I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.

原话

I like this post. I haven't had time to dig into Jev (they aren't accepting new signups), but calibrated probabilities is one of their pitches that caught my attention. And I was wondering how does one offer them on user data. Standard calibration essentially ensures that if a score of 0.8 accompanies a positive prediction (assuming the simple case of binary classification), then if you gathered together all predictions with a score of 0.8, around 80% will be correct. If you have just one example you're sending to a model, how would they guarantee 80% over your data?

原话

Article says that it _can_ be calibrated, it just isn't out of the box. So the title seems contradicted by the body.> If you want calibrated probabilities you’ll still need to recalibrate Jev’s probabilities on your own data. The good news is that is cheap. A few hundred labeled examples from your actual data can be enough to fit a Platt scaling on top of Jev’s scores.