The scoring system in slopscore has been completely re-done!
First: The way signals are tracked stays the same. If a repository had 25 AI-generated commits detected before, that'll still be just the same. The only thing that was touched was the way the score (100 (S)–0 (F)) gets calculated.
Instead of treating every detected AI/slop signal equally, slopscore now tracks the share of tainted commits across a repository.
This means: It now makes a significant impact whether the repository is made up of slop commits to 1% or, say, 50%. Repositories with lower slop-densities get gentler penalties, while high-density repos get scored significantly harsher. It just didn't feel right that 10 slop commits got scored the same, whether the repo only had 15 commits, or already had 10000 commits.
This feature has been in beta for a while and the new scoring curve has been tested for a while before it finally got merged and deployed today.
Do feel very free to check out the changes by re-scanning repositories you are curious about, and leave feedback either here, on Zulip or as an issue on the repository!
@polyphonychat Thanks, such a great tool! I will say its a tough one but slopscore.ava.pet/repo/https://github.com/kanidm/kanidm the agents.md is there to stop ais working, but did cause a penalty for us. Hard to know how to fix that though? Perhaps if agents.md has the refusal string then it doesnt cause a score loss?
@firstyear Thank you very much for the encouragement! c:
Yeah; we want to tackle this problem somehow, but we are not quite sure on how to best do that yet. It's something that is being actively discussed right now, and that we are hoping to solve some time soon :)
@polyphonychat@firstyear a magic char sequence to indicate it's an anti-AI file? The projects that care about such things probably wouldn't mind a small hoop to jump through, and I doubt the boosters would bother to game just that one signal, at least on its own?
@polyphonychat Было бы классно сделать какой-то рейтинг или топ репозиториев которые погрязли в ИИ слопе. Сейчас на странице просто списки репозиториев и непонятно кто из них больше кто меньше, возможно стоит мерить % ии кода от общего процента кода репозитория.
@markhughes@polyphonychat it's realism, I guess. For lots of things that everyone is going need, e.g. kernels, there are no realistic slop-free alternatives right now. Being able to choose the relatively less contaminated stuff feels like a win of sorts, albeit in a very depressing way.
It does occur to me that that older projects that existed decades before the advent of AI are preferentially advantaged, though. I wonder if only looking back a maximum of a couple of years would be worthwhile?
@magnetic_tape That's a good idea and something we can do. What might help in the meantime: The score also has a tier from S-tier for "best" to F-tier for "worst" alongside it. Though I understand that not everyone might be familiar with this system.
@hexa Yes. For this ratio, penalties are being reduced to 4% of their original value. While that is a 96% reduction, this still stacks, of course. The rating is as expected.
