At the core of our service is an RSS feed reader, which acts as our aggregation tool. The feed reader is intended to be very low impact and somewhat intelligent, as not to impact your site in anyway. Below, you will find specifications for our feed reader so you can familiarize yourself with how it operates.
- The feed reader checks an RSS no more then once every eight hours. This is true regardless how many of sites import from a particular feed and regardless how big the feed is.
- The feed reader respects robots.txt definitions. Every time it updates a feed, it downloads two files: first the robots.txt information and then, if there are no rules preventing the importing of the feed, it will proceed and download the feed. If you would like to stop crawling, you may add a rule referencing the user agent string "MaxMoFeedReader". For more detailed instructions, see the page on removing content. Once our feed reader has been blocked, it will never automatically check your site again (it disables itself). If you would like to once again allow importing of your feed, please contact us: content@maxmo.net.
- If the feed reader is unable to get through to your feed or unable to parse your feed, it will keep trying every hour for eight hours. Should it continue to fail after eight hours, it will disable your feed and not return again.
- When importing articles, feed reader makes minimal changes to ensure the clean and consistent visual style that we strive for on our sites. These changes are minor, but are simply meant to ensure that articles appear in a clean and presentable format. Changes include: removing JavaScript, removing certain tags (tables, frames, iframes, etc) and removing images (we may start including images again in the future).
- Thanks
The MaxMo Content Network Team
