I wrote a simple crawler in C# last night. No rocket science involved. The solution drills through a site looking for specific terms. When the crawler finds a match it archives out the source. The crawler hits a root uri, then parses the HTTWebResponse stream resolving all links, then the crawler recursively searches it’s way through each descendant link. The code is crying out to be threaded which is the next step.