C# Crawler

I wrote a simple crawler in C# last night.  No rocket science involved.  The solution drills through a site looking for specific terms.  When the crawler finds a match it archives out the source.  The crawler hits a root uri, then parses the HTTWebResponse stream resolving all links, then the crawler recursively searches it’s way through each descendant link.  The code is crying out to be threaded which is the next step. 
This entry was posted in .NET. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s