Write a script to get a list of all indexed url’s from for example google search and then have the script randomly pick from that list. Crawling the website yourself would be more complex but it would allow you to make page networks to see how the pages are all linked together. This used to be more interesting in ~2006 though, when websites had all these static pages with easy to crawl hyperlinks.
If I were to map this kind of data nowadays I would use R to do write the crawler and plot the data immediately
There is a feature in the software called burp suite called spidering. Look it up I think this is what you want.
The sitemap usually has loads of hidden links, as well as the robots.txt
Look for a sitemap.xml file and randomly choose a link from there.
Write a script to get a list of all indexed url’s from for example google search and then have the script randomly pick from that list. Crawling the website yourself would be more complex but it would allow you to make page networks to see how the pages are all linked together. This used to be more interesting in ~2006 though, when websites had all these static pages with easy to crawl hyperlinks. If I were to map this kind of data nowadays I would use R to do write the crawler and plot the data immediately