Google crawler on protected pages

Hi, I wonder how google crawler deals with protected pages.

I would really like the spider could crawl the entire page, regardless the page is protected or not.

Some time ago, I read somewhere that the spider doesn’t consider javascript, and as long as memberstack protect the pages with a js based solution, it should be able to scan all the content.

Can you guys give me some more infos?

Thanks
Francesco

1 Like

Hi Francesco :wave:

In our experience, Google’s bots respect Memberstack’s redirect code and do not crawl the page.

I’ll need to keep thinking about this one. So far, all of our mental energy has gone into keeping Google OFF the page.

Can this snippet be the solution?

If I detect that the google bot is currently reading the page, I could avoid the data-ms-content attribute (I’d like to protect only some content of the page)

1 Like

It’s certainly worth a shot!

We’ve had a hard time testing these sorts of things reliably given that Google’s bots don’t always behave in the same way. If you have a few pages that rank for known keywords I recommend a quick A/B test.