If you're not comfortable just say "we don't do sleepovers yet" (family excepted). If you just state it (rather than make an excuse iykwim) I'm sure people will just say "okay" and not worry about it. A couple of my son's friend's families have that policy, and we never feel offended or put out. Each to their own!
Our children's first sleepovers where young - DS1 would have been 4 or 5, and DS2 was 2 1/2 and still drank milk at night. Plus we all slept in one big mega bed at that stage. But it was both of them together, with very close friends who lived just around the corner, and all four children slept in the same room and my two even slept in the same sleeping bag! They loved it!
They then had a group of about 4 or 5 families who they (and us!) felt comfortable having sleep overs at, and their children loved coming to ours for sleepovers too. Often it worked for the parents too (a night out!) and often it was siblings. One of DS1's friends had two or three attempts before he managed a full night - but he really really wanted to, and no-one minded the "I want to go home now" at 9pm LOL. Never had drama or tears or anything negative.
When DS1 was 7, he spent from 3 to 5 nights (he went about 4 times) with a friend who had moved 2 1/2 hours away. That was hard for me, but I never showed it

he was ready to do it, we knew the family well and were totally comfortable, and it opened up fabulous new experiences for him.
Something I often chuckle to myself at is how you can tell who cosleeps at home and who doesn't - some of their friends all want to bed down on one mattress, other's need their own mattress and own space or they can't sleep.