No idea whats going on with my bot, however i have some ideas since i've rewritten the entire members core so that a user is not remembered by their nick but their ID, thus i think its messing up with the friendslist somehow.
So my solution was this:
bot.FriendList.RemoveAll();
Then restart bot. Dirty but it was only way i was able to wipe / remove any "friends" and removing then from notify did no good as users where still inn my "recently spoken with" category, which also counts towards friends, silly but that's just how it is.