Here is what a robots.txt file could look like. It is just a plain text file in your root directory.

User-agent: *           # directed to all spiders, not just Scooter
Disallow: /images
Disallow: /pubs/images
Disallow: /lesson
Disallow: /cgi
Disallow: /directory
Disallow: /Docs
Disallow: /gifs
Disallow: /imagemap
Disallow: /Lesson_Scripts
Disallow: /NCPages
Disallow: /WWW
Disallow: /pubs/cgi
Disallow: /pubs/comments
Disallow: /pubs/pubsemail.html
Disallow: /pubs/addbook.html
Disallow: /pubs/crawler.html
Disallow: /pubs/footer.html
Disallow: /pubs/index.html
Disallow: /pubs/index.shtml
Disallow: /pubs/indexwait.html
Disallow: /pubs/list.html
Disallow: /pubs/oldindex.shtml
Disallow: /pubs/pubcrawl.html
Disallow: /pubs/pubsbook.html
Disallow: /pubs/random.cgi.rnd