I have a website that returns these two lines
User-agent: *Disallow: /
I built a regex to recognize this. It looks like this
User-agent:\s\*\sDisallow:\s\/
I tested it hereAnd it seems to match it.
I setup a content monitor with this regex string to detect this but it does not work. It reports trouble. I can find no documentation on your site about the regex feature or how it should be used. Can anyone help?
The actual site is https://or-beta.nmdp.org/robots.txt
You are free to browse that link or make a test on it yourself if you like. Just remove your test when you are done so we don't see your traffic long-term.
Thanks for anyone who can help.
Like (1)
Reply
Replies (1)
I got it to work by looking more closely at the actual output.
The
expression that works is this
User-agent:\s*\*\s*Disallow:\s*\/
My mistake was using a single whitespace character "\s"
when I needed to allow for 0-N whitespace characters.
So using "\s*" was more flexible.
Like (0)
Reply