There were about 1.4K albums with mean plays greater than 1. From this import, I reasoned that I can grab the albums where I have listened to it more than once and that will do as a recommendation. I have a method for importing my iTunes/Music library as XML into R (I plan to write this up in the future). So this is where the build starts… Use R to make a list of recommended albums I wanted it to post albums that I would recommend. The idea is just to put an album suggestion into people’s feeds so that they might be inspired to hear something new or revisit a classic album. The toot is simple the artist, title and year, include the cover, and… that’s it! People can leave a comment if they love the album, discover something new or whatever. The idea is to periodically post an album. You can follow the bot – currently called Albums Albums Albums (or AlbumsX3) – here. There was no mastodon equivalent and so I decided to build one. Setting up a robots.txt file that looks like this: User-agent: *Īllows all user agents to crawl all pages on your site, but as this is the default behaviour anyway it might be better just to leave it out.I have long admired albums2hear, a Twitter bot that posts albums. In my personal opinion I would only use a robots.txt file if you have anything that you want to prevent being crawled or you want to add a link to your XML sitemap. What was more interesting was that many sites simply didn't have a file called robots.txt, which leads me to think that they aren't all that useful. A few used the file in order to point at their sitemap.xml file, which was usually called something else. I had a quick look at lots of sites that are run by people in the SEO industry and found that many used the robots.txt file simply to disallow certain files or directories. Here is a robots.txt file that references two sitemap files that each contain half of the site. You can link to a sitemap index file (which contains references to other sitemap.xml files and might not be called sitemap.xml) or simply link multiple sitemap.xml files from a single robots.txt file. The power of this option becomes apparent when you have a large site and need to spread your sitemap file across different files. If this is the case then a rewrite rule might be better, but it is still possible to to this with your robots.txt file. You would normally use this option if your sitemap.xml file was created by your CMS and has a non-standard name. This line is a little redundant as all of the search engines above will automatically look for a file called sitemap.xml situated on the root of your domain.
You might add the following to you robots.txt file. Along with this format is the ability to add a link to your sitemap.xml file from your robots.txt file. Google, Yahoo, MSN and other search engines have adopted the sitemap.xml format. So is it beneficial or even useful in terms of SEO? Well there is one use of the robots.txt file which can have beneficial results, although how beneficial depends on how big or complicated the site is. There are a couple of reasons why you might want to stop this from happening, but here is the rule you would need.
In the following example we are stopping a user agent called ia_archiver, which is used to create a copy of your site at. In another scenario you might want to stop a certain bot from crawling the content of your site. You would then put a robots.txt file in place with the following content. This stops is appearing in search indexes and things so that you can get your users to grab the Feedburner feed and not your local feed. Once you have allowed your blog to issue the Feeburner feed instead of your normal feed you then need to stop search engines from indexing the old feed. I won't go into why you would do this much, other than to say that you get some nice usage statistics and it can save on some processing power on your server as your feed is only looked at when Feedburner updates. Lets take a real-world example and look at what you would do if you decided to set up a Feedburner feed in place of your normal RSS feed.
You can even prevent certain user agents from crawling certain areas of you site. A robots.txt file is a simple, static, file that you can add to your site in order to stop search engines from crawling the content of certain pages or directories.
0 kommentar(er)
