Slightly Shady SEO | Blackhat is Back, Baby.

I know this is a long entry, but it’s damn useful, and can bring you up to the next level. Whitehat or Black.

As promised yesterday, this entry is going to discuss how to take your already existing SEO setup, and turn it into an automatic machine. I have never heard of a 100% automated system, but this is the ideal. Even automating individual sections is incredibly useful. When designing individual parts, build them with future integration in mind. Use the same databases, the same basic systems, and have easy ways to pass information and variables. Eventually, by just automating individual components, you may eventually have inadvertently built a system similar to this one, and then it’s simple work to bring it all together!

This is primed for a

The Optimal System/Parts

1. Keyword Selection
   1. Have a “queue” of future topics, so you never lose any time switching from one to the next. MySQL databases are best.
   2. Automate your keyword selection. I won’t get into where I get my keywords, but wherever you get yours, have it automated.
   3. Create a new new table for each topic, and then the various keywords associated with it
2. Template Generation
   1. It really is best to download free templates, then modify them so that they’re easy to substitute data into. Mark each link with a keyword, where the content should go with another, and the same for titles/subtitles. Replace those later with the needed data.
   2. If you’re too large scale for the above idea, randomize the templates. Numbers of columns, images, locations for each element, all that. Also, do not overlook randomizing form names, and where the line breaks(relative to the code) occur.
   3. NEVER have a piece of text be static. There has been situations where Google knocked out a blackhats entire site list, regardless of hosting/template, because of some footprint that went overlooked. Also, it allows other blackhats to mine your tricks.
   4. Consider scraping flickr for relevant images. Depending on how you formulate your search, you can get them cleared for reproduction/commercial use. A nice touch. Also, the images are almost always on topic, and quite high in quality. Google images will give you a lot of site formatting images, thumbnails, and other such uselessness that can be a giveaway for an auto-generated site.
   5. If your templates are not 100% generated, and is made using free online templates(or those designed by you), make sure you have an easy way to enter the templates into the rotation. A quick zip/tar.gz uploader is best. Also, that way, if you pay someone else to create your templates, that person can enter them in to the rotation with little to no intervention by you.
3. Content Creation
   1. Know your approach. You really have only 4-5 options
      1. Direct Scraping, full data.
      2. RSS Feeds
      3. Content Generation/Markov Scripts
      4. Manual, offshore labor
         1. Make sure to have an easy way for those who do your writing to retrieve their assignments. Get a reliable crew that will check the buffer every day, and start pumping out the desired articles. Include an easy way for them to submit their work on a webpage.
         2. If possible, have an automated payout system. Keep an automatic tally of their submitted articles, and have your script login to paypal and send them their payment. Be careful though, to avoid no payment, or god fobid duplicate payment.
      5. Gibberish (Scrape/Cloaking sites)
   2. No matter which way you choose do get your data, make sure it’s stored in a swiftly accessible database, and backed up consistently. Have it so all sites that are out there reference this database by domain, not IP. This way, if that server goes down, or is too distant from your most active web host, you can easily re-reroute the traffic to the backup database.
   3. Have your content creation feature tie directly in to your keyword/topic database.
4. Site Creation
   1. Obviously, this requires a tie-in with the keyword scraper, the content creator, and the template creator.
   2. Have automated checks to ensure all data is valid and present. You don’t want a junk site to appear, with no content due to a failure way back in the process.
   3. Have a zip file stored that has all your scripts that you use in every site. Your creation script can extract these, and copy them over to your new site. This would include logging scripts, cloaking scripts, perhaps the script that inserts the data into the template. Also, this is a good place to store all of your templates.
   4. Write a CURL script that can add domains/subdomains automatically. It’s a good idea to create the subdomains based on the “category” you pull from the database. Make sure that your script checks the output of cPanel, to ensure that the creation worked. Optionally, you can also have your script visit the subdomain, to make sure it’s pointed properly and resolves.
   5. Another script should check to ensure you don’t end up using a banned domain, or one that’s overly saturated. More about this later though.
5. Promotion
   1. Link Spamming
      1. XRumer has the ability to pull its links from a text file. Use this to your advantage. That text file(I believe) can be modified on the fly.
      2. XRumer also comes with a scheduler. Learn to love it. This can also assist the Google penalties that come from sudden link surges(schedule, say, 1000 posts per day per subdomain)
      3. Make sure you don’t hit the same message boards/blogs/guestbooks every time. Keep it shuffling. Use blocks of 50,000 links per file, and randomize file selection/start point.
   2. Blog Farms
      1. I won’t get too in depth here, because certain people would be angry. But suffice it to say all your posting/pinging/data gathering should be seperate components of an overall system.
      2. Integrate this with everything else. Within 48 hours of site creation, set a certain % chance a blog will post an ad for your site. Also, randomize which link/domain gets advertised. These both help protect against getting detected as a link farm.
   3. Directory Submissions
      1. If you’re using a combo approach, do this BEFORE link spamming.
      2. If possible, use directories and a few blogs to get indexed, then follow it up with link spam if your program detects you’re not at the desired position in the search engines.
      3. Remember, this needs to be automated. No manual submissions. It’s not too hard.
   4. Domain Registration
      1. Monitor this carefully. It could run up a bill otherwise.
      2. Have a list of words that you can append to most topics that would make sense (super%topic%resource.tld for example), or, if you’re operating largely of subdomains, a list of generic words that can be combined to make decent sounding domains. For example (web/internet/web)(data/information/info/knowledge/fact)(database/archive/resource/resources). Type these macros out, and allow it to randomly check to reserve them.
      3. Remember to also modify the DNS servers as necessary,and add them to your web host. This could require a cron job, since domains take time to propogate.
6. Tracking Your System(THIS ONE IS IMPORTANT)
   1. Any errors that occur should be sent to a mysql database archiving the errors. If this is not possible, then drop the errors in an e-mail or text file.
   2. All scripts should check their input to ensure that the data being passed is proper, abundant, and valid.
   3. There are 3 types of Errors that can occur. Track each differently.
      1. Negligible Errors - Site that you were scraping doesn’t exist anymore, perhaps you didn’t get quite as many keywords as planned. Worthy of a mysql database entry in the errors table, but that’s about all.
      2. Step Failures - Something goes awry during some step. Data doesn’t get scraped, a domain doesn’t get added properly, things like that. These will stop NEW site creation until fixed, but will not disable your already existing sites. These should get an e-mail sent to a frequently checked account if it’s a one time, or occasional error. If it’s completely disabled, send an e-mail direct to your cell phone
      3. Point of Failure Error - A server goes down. MySQL gets compromised and a password changed. Domains aren’t resolving. All of that.
         1. Attempt to build in an adaptive system that can fix itself. Re-route domains to new IPs, begin use of the backup mysql server, things of that nature.
         2. These ALWAYS get an e-mail to the cell phone. Always. If it can be auto-fixed, follow up with a message saying so, but still, get your butt online and fix things. No backup server should ever have to be used for more than 6 hours.
   4. Monitor Your Servers. Even if it makes the stats messy, use however many servers you’re using for this setup to constantly connect to eachother with different ports. If a problem is detected, have one of the servers(just one of them. A chain of command is useful here) attempt to fix the problem by re-routing whatever is necessary.
   5. Monitor your Sites
      1. Know which domains are banned, and what their topics are. Add any banned site that made more than it cost to the queue to be created again.
      2. Know how many pages were indexed yesterday, the day before, and the day before. This alerts you to bans.
      3. Know the ranks of your sites. Don’t promote when it’s not necessary. If your site is ranking where it should be, certain promotional steps can be bypassed. If the site is dropping in rank, your system should add it to a queue to be promoted.
7. Tracking Your Income
   1. Do not sacrifice versatility in favor of the ability to track income. You should be using multiple affiliate programs, in multiple niches.
   2. Use a Curl script to login and grab your income. Write this module-style. A different module for each different piece of affiliate software.
   3. Track how many sites you have in each niche, and what the income from that niche is. If something is making an especially large profit margin, add it to the queue to be duplicated.
8. Miscellaneous Notes
   1. This is obviously written from a blackhat perspective, but much of it can be ported to a whitehat setup for rapid site generation.
   2. Never ever hardcode database names, server IPs, or domains into the program. These should be stored in a settings.ini file. If you need to pickup and move, it’s a pain in the ass to run through a script changing everything.
   3. Frequently done functions should be stored in a library accessible to all scripts. This makes life easier, errors less common, and issues easier to diagnose and locate.
   4. Just because it’s automated, doesn’t mean you shouldn’t check in on it. In the early days especially, watch it, AND the sites you’re making, like a damn hawk. Pay special attention to payouts, domains, and anything else that involves you spending money.
   5. Important: Avoid having scripts directly call eachother. A properly implemented queue system should make it so you don’t ever need to directly pass data from one script to another. A central controlling program(PHP script if that’s all you know, VB/C++/Java otherwise) can call each script on a chronological basis, and is much more efficient, and much more reliable than a web browser(wouldn’t want your whole system to deadend because Firefox locks up…again).
   6. Start out slow. Create your databases before your scripts. Put your layout down on paper. Know what’s getting data from where, and where it’s putting it. Then create individual components. Preferably, make sure that the individual components can function with your intervention, before the completion of a system. That way you have money while you’re building. Also, nothing finds glitches like real life testing. A massive system, implemented all at once, is almost sure to have vague and untraceable errors.
   7. Make moving servers easy; don’t be lazy. Make sure your table exists. Make sure the database exists. If they don’t, make them! Optimally, if you move the scripts from one server to another, you should be able to start the system without modifying a single file(aside from DNS ip pointers)
   8. Run a sandbox for updates, before putting them live. Nuff said. It can even be on the same server, just with mysql/apache running on a different port.
   9. Spread out your work load. It’s worth it sometimes to have the server doing your scraping for example, separate from your command and control server with all your databases.
   10. Make sure your shit is secure. Unique mysql logins for each server, properly configured software, password required for manual interference.
   11. Allow for manual interference! You should be able to insert your own niches/keywords. Manual intervention can sometimes turn an ok site, into a spectacular site.

I know this is a lot to read. Really, I do. I feel bad for you guys. But it’s worth it, and is sure to get your brain moving. Think big people!

Posted on Friday, November 9th, 2007 at 7:09 pm in the category:automation, seo. Comment RSS 2.0 feed. Comment it , or trackback this post.