A lot of this is down to changes with Google, I think. User-generated content, for want of a better word, are quite down-ranked. I can have another look at it but I got the impression that bringing in new users and promoting the forums wasn’t a priority.
Since we moved to self-hosting there’s definitely been a big drop in traffic and issues due to search engine spiders.
Anyone got any bright ideas?
To add, I feel that it’s not just low ranked. I’m pretty sure it’s literally impossible to google anything on this forum, no matter how far down the results you can go. Happy for someone to prove me wrong though.
I think it’s nice that we have a close knit community, but I don’t think we’ll get a huge influx of users as such as to change the vibe of the place, and it might die out eventually if no one new ever finds the place. I remember there was real surprise that the forums were still going when you posted the thread about DiS on r/indieheads.
Unfortunately I’m not technical enough to help with the ‘why’ part, but hopefully someone else will know…
I’ve just tried resubmitting it and got this
Failed: Blocked due to access forbidden (403)
is there a line of code stopping it, maybe?
The setting is on in the back end
But the Robots Txt says otherwise https://community.drownedinsound.com/robots.txt
Any ideas @admins ?
Have all the steps listed here been followed?
So all of this was for nothing? 
robots.txt and sitemap.xml both look fine.
The issue is with nginx - it’s blocking requests from Googlebot…
I’m assuming the robots txt is blocking AI training bots?
Okay, replicated on my laptop and fixed. At some point “mozilla” was added to the list of disallowed user agents for web crawlers.
Unfortunately, basically every bot access the site uses that at the start of the user string. E.g. Googlebot identifies itself as Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.7137.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
It’ll take some time for Google to recrawl everything, but it’ll reappear over time now.
Not at the moment - it’s rate limiting them to one request every ten minutes though. I can add those to the ban list though if you like?
Amazing, thanks so much
That’s incredible, well noticed! Great work @zeal!
Damn I was enjoying the forums being completely absent from Google. Everyone get dressed.
Yeah, here you go - it’s already started crawling.
https://www.google.com/search?q=site%3Acommunity.drownedinsound.com
@xylo ‘s trip to ‘Merica looking increasingly precarious
We can make all but the music forum private, feels like it makes sense?
Is your real name connected to your account in any way? Would he really hard to Google you
They’re more likely to ask for your phone and see what you’re logged into
This change was mainly so that people can use search to find their own topics using site search - as our search doesn’t always do it
I think @moderators might have a solution for this?
Was this meant to read this sinister?
