The fee structure is both a filter against superfluous submissions and a revenue generator. Typically, the fee covers an annual subscription for one webpage, which will automatically be catalogued on a regular basis. However, some companies are experimenting with non-subscription based fee structures where purchased listings are displayed permanently. A per-click fee may also apply. Each search engine is different. Some sites allow only paid inclusion, although these have had little success. More frequently, many search engines, like Yahoo!, mix paid inclusion (per-page and per-click fee) with results from web crawling. Others, like Google (and as of 2006, Ask.com), do not let webmasters pay to be in their search engine listing (advertisements are shown separately and labeled as such).
Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
In addition to measuring your website traffic, you need to track your conversion rates. For example, if, as a result of your search engine marketing efforts, your traffic doubles from 250 to 500 visitors per month, how many new customers did you acquire from the additional 250 visitors to your site? Do you now have twice as many customers as you did before? If you picked up 5 customers your conversion rate would be 2 percent of the new traffic (5 divided by 250) and 1 percent (5 of 500) overall.