Raspberry Pi from Three and Three Fourths

Finally we are at the end of setting up our magical torrenting client and all we need to do is have a way to add our anime to an rss watch list. Luckily enough, we have just the plugin that can deal with RSS feeds, but to keep things in order, first we need to decide on a feed to follow.

The normal sites that I assume are standard when it comes to searching for this kind of media: Nyaa.se, Tokyo Toshokan, BakaBT, and ShanaProject. Out of these, ShanaProject is probably the most anime-oriented and is kind of specifically a site just for tracking anime and has user specific feeds that you can use to follow and track different fansub groups for different anime. It's the default feed of choice for me, but there are perks to using the others as well. Something to note though, Shanaproject along with Tokyo Toshokan pretty much aggregate torrents from Nyaa (at least in terms of anime), so there isn't much point to using all three of them.

To give a quick run-down of the options aside from ShanaProject:

  • BakaBT: A "community" that dislikes leechers and doesn't seem to have any RSS feeds, making automation a bit of a bitch.
  • Nyaa: The defacto tracker for pretty much all of the fansub groups. If they aren't on Nyaa, they are just retarded somehow.
  • Tokyo Toshokan: The best option for all types of media from Japan. It aggregates anime, manga, music, etc. and has a customizable RSS to boot. Also, it has some re-encoders (hi10ani.me) if they actually put up torrents.
Anyways, after picking a feed to follow, we just click on the nice little rss icon in rutorrent, add our first rss feed and give it a name. Clicking on the rss icon afterwards now will bring up the RSS manager where all things great and automated occur. It's pretty simple from there, just come up with a name for your new subscription (usually I just title it the name of the anime), a regexp pattern to look for (ex. "/Akame ga Kill.*\[FFF"), the feed to watch, and a subfolder to download it into if you want (it defaults to the downloads directory of rtorrent). After that, the ratio rule and group from the previous post will deal with removal.

At this point, if you added the anime for each running season, your setup would automatically download, seed, and remove your torrents for the entire season without you needing to do a thing. It is probably possible to add an extra script to automatically generate even the configurations for the rss, but I currently don't have a script for that (in due time, I will get that since there happens to be a rather nifty link available on MAL).

To take rutorrent a little bit farther and make it actually decent as a torrent client for things like older anime series or episodes that are not on the latest feeds of Nyaa or Tokyo Toshokan, I made some extra search engines for the extsearch plugin (thanks to the wonderful and fairly simple examples of the other engines). Just save them into /srv/http/rutorrent/plugin/extsearch/engines/<searchengine>.php files and they should magically appear and be ready for use.

Nyaa: http://pastebin.com/D5S6NCyi
SukebeiNyaa: http://pastebin.com/fLgXAAxu
TokyoTosho: http://pastebin.com/mS67ZqnX

Anyways, with that we FINALLY have all of our torrent client/rss aggregator woes fleshed out and out of the way, but we still have one little extra that we can do. After setting up auto ssh authentication (this can be found anywhere on google) from the raspberry pi to your obviously linux-running computer, you can make a rtorrent_notify.sh script to send a notification or something to your computer for when a torrent completes. My script is really simple:
ssh user@host 'DISPLAY=:0 notify-send "New episode ready!" "$1"'  


Raspberry Pi from Three and a Half

So with our rutorrent and rtorrent set up, we can now move on to tackle more first world problems and start this stuff on boot. The main goal is to get lighttpd and rtorrent up and running when the pi starts up, so that we can instantly go to rutorrent after it finishes booting, and then do a little bit of configuration to rtorrent and rutorrent to smooth out some of the edges.

First is to install tmux or screen, though which one in particular is a toss-up (and I prefer tmux). After that, we need a systemd service to start tmux/screen and rtorrent, as well as stop it. Of course the arch wiki has a nice entry that you can pretty much copy/paste, though I like keeping the rtorrent as the full name for things rather than rt. In any case, "sudo systemctl enable rtorrent@user.service" and similarly for lighttpd.service will start both at boot so you don't have to do that manually and you can check your rtorrent by ssh'ing in and opening that tmux/screen session.

We are going to be taking a look at the .rtorrent.rc file to add a couple of extras that we'll need and other things. As a warning, pretty much all of the configurations in the .rtorrent.rc are also available in the Rutorrent webui, but for whatever reasons (maybe some permissions problems somewhere) it doesn't seem to save those configurations across reboots. So things like the download directory (best to set to the highest folder in the hierarchy of your anime), session directory, port range, port random, encryption, etc. should be set using the .rtorrent.rc file (and you should set them). At the very end of this file, we are also adding two lines:

execute = {sh,-c,/usr/bin/php /srv/http/rutorrent/php/initplugins.php &}
system.method.set_key = event.download.finished,notify_me,"execute=~/.cache/rtorrent/rtorrent_notify.sh,$d.get_name="

The first line is to load the plugins for rutorrent when rtorrent starts up so our feeds start up. The second line on the other hand is COMPLETELY OPTIONAL and is pretty much just executing a little script called "rtorrent_notify.sh" after a torrent finishes downloading. Supposedly with a little magic and some bash, you can send a great little notification to your computer from your raspberry pi that tells you when a new episode is ready.

The rest of the configuration can all be done with rutorrent and some time should really be put to exploring all of the settings available, but there are some quick things that I will highlight:
  • General>User Interface>Theme (the oblivion or dark themes look pretty nice)
  • Ratio Groups>Any number
The first bullet is just a preference thing, but the second is half of the staple way of automatically removing torrents from rtorrent when it's finished downloading. It's fairly easy to understand, just name the ratio to any name, choose a minimum percentage that you want to seed back to your peers, maximum doesn't really matter, a minimum amount in MBs that you want to seed back, and for how long (or just ignore the time all together), and then set the action to remove. If you wanted to be a real asshole, you could set all of it to zero and then as soon as your file finishes downloading, it gets removed or you could be a decent person and seed until you have given as much as you have received. To complete this automatic removal, we need to make sure that every torrent gets set to this ratio, and we can do that with the Rules Manager (button right next to the help, second option).

Pretty much just make a new rule where "if the torrent label contains <blank>", set the ratio to your new ratio. And by blank, I literally mean just a blank entry. With that, almost everything is taken care of in terms of our torrent client and we can look at the way we are going to be automagically adding new torrents to our heavily under-powered torrent box.


Raspberry Pi from Three and a Fourth

So getting rutorrent onto a raspberry pi is not nearly as amazingly easy as deluge, but hopefully it will be better in the end. Since a lot of the typing work has already been done, I'll just link to the arch wiki which should get you most of the way there. Just make sure to copy the template .rtorrent.rc to your home directory with "cp /usr/share/doc/rtorrent/rtorrent.rc ~/.rtorrent.rc".

After following the wiki, it brings you to the point where things almost fully work, but the log of rutorrent (when you open it in a web browser) gives you some of the issues it encountered.

The first of these might be something along the lines of the rtorrent user needing to have read/write permission to the tmp directory. Now I'm no expert in dealing with permissions, but to make things simple and quick to fix, just make a new directory in the /tmp folder with "sudo mkdir /tmp/rt" and then change it's permissions with "sudo chmod 777 /tmp/rt". Finally, taking a quick visit to /srv/http/rutorrent/conf/config.php, at the very end there is a line that says "$tempDirectory = null;" which you change to "$tempDirectory = '/tmp/rt';". Instantly problem solved.

The other issues that probably got logged were about the plugins. Some of these plugins you might want to disable (like the rutracker_check one), and if you want the other plugins (mediainfo, screenshots, unpack plugin), just install the packages that they are asking for (mediainfo, ffmpeg, unzip, and unrar respectively). The mediainfo, unzip, and unrar packages are pretty small in comparison to ffmpeg, and have their uses in other places as well...ffmpeg on the other hand is about 120MBs, but I'm still going to probably install it anyways. Now is also a good time to take a look through all of the plugins available in rutorrent (all the non-3rd-party ones anyways) and disable unneeded ones. It's not a huge hit in performance leaving them on, but meh, I'll take anything to lighten the load.

Now these problems are only what rutorrent can show you. There are more problems hidden in the /var/log/lighttpd/error.log that you can also spend time fixing :D. Most of the fixing is going to be in the /etc/php/php.ini, so open that up somehow and we want to add some extra directories to that open_basedir line after "/usr/bin/" to get our disk space plugin to work. Just change the end to "...:/usr/bin/:/:/usr/local/bin/:/usr/local/sbin/" but don't close the config file just yet.

Next is to fix the issue that get's spammed into the lighttpd error log about the time not being set. If you are using nano, use ctrl+w to open the search function and search for "date.timezone". This will bring you to the setting in the config where you want to uncomment the ";date.timezone =" and add your timezone to the end. In my case, I change it to "date.timezone = America/Los_Angeles" and the value for you is just the same as when you set up your timezone on the raspberry pi. After that, you can save the config, kill any php-cgi that might be running, maybe reload the systemctl daemon, restart lighttpd and give your rutorrent install another check.

Finally, we can clean up the horrid mess we made of everything and make things a bit nicer with starting at boot.


Raspberry Pi from Three

So with a great file system on your hard drive, we need to do a quick addition to our /etc/fstab to mount it at boot, but first is a recap of the other stuff we still need to do.

  • RSS aggregator of some sort to sift through the torrents we are looking for and grab the ones we want
  • A torrent client to actually get them without killing our pi
  • A protocol to serve our media to another device on the network
  • Maybe something to hide ourselves so we aren't blatantly telling the world that anime is all we ever want

  • Now to mount our drive at boot. Just pop open /etc/fstab with something like "sudo nano /etc/fstab" or any other preferred text editor. You will need to know where your block device is (/dev/sda1, /dev/sda2, /dev/sdb1, etc.) and just slap an entry along the lines of 
    /dev/sda1     <some mount point>   <file system type (ie. ext4, ntfs, )>         defaults       0     0
     And with that, we can finally continue towards tackling the next major issue on our list of things to deal with, the RSS aggregator.

    There is a small number of programs at our disposal that we can use to deal with this problem, with the most notable ones being sickbeard (not really) and flexget. After that there are some others like rssdler, couchpotato, etc. that I didn't look into. Aside from these, there are also some RSS aggregators that are included (usually as a plugin) in torrent clients like deluge or rutorrent (frontend to rtorrent).

    Sickbeard's support for anime is shoddy at best, with the main way being a branch of sickbeard that isn't really maintained, making it kind of pointless...and while flexget is better, I just never liked flexget and so I didn't try it. Rssdler might have been a good choice, since it looked pretty lightweight to match rtorrent, but after 5 years of next to zero development activity, it's pretty much a dead project.

    Thus, I really looked towards deluge and rtorrent as being the grand solution. Setting up and preparing deluge is amazingly easy and primarily relies on a plugin called YaRSS2. Simply install deluge, start the daemon, connect with a deluge client, and install the plugin. After that, you are just about ready to start reading rss feeds and auto-downloading the torrents. Since the wiki is probably loads better than any explanation I could pull off (it has examples), the link is here. Rtorrent on the other hand...has you going through a couple of hoops, but in my opinion, those hoops are just barely justified.

    When using deluge, you would need to use a client and while there is a plugin for a webui, it unfortunately doesn't have a webui for YaRSS2 which is our primary way of automating our anime downloads, meaning we would either need to utilize a script which would add a new entry into the YaRSS2 config after we type in the anime title somewhere or use the gtk client. At the point where you are going to just use the gtk client to connect to the deluge server, you might as well just set up the server on your computer and use deluge in classic mode. Rutorrent on the other hand has an RSS manager plugin and of course, since the whole frontend is a webui, the plugin also has a web interface. This means that when you check your Raspberry pi's torrents, you can manage every torrent-related thing from a single webpage without needing a client (managing it on-the-go).

    Since setting up rtorrent+rutorrent is a bit of a journey, that will have to be in the next post.


    Raspberry Pi from Two

    So continuing the thread of the raspberry pi, we currently have some magically way of managing our raspberry pi via ssh.

  • A storage place for the stuff we are hoping to use our torrent client for
  • RSS aggregator of some sort to sift through the torrents we are looking for and grab the ones we want
  • A torrent client to actually get them without killing our pi
  • A protocol to serve our media to another device on the network
  • Maybe something to hide ourselves so we aren't blatantly telling the world that anime is all we ever want

  • So now things come to how we are going to store our data and because of how completely dense this topic is, handing out a singular solution would probably not help at all. Fortunately, there are some things that could be solidified even if you decide to pick and choose some other bits and pieces.

    First off is the storage medium, which is to say the hard drive, usb stick, ssd, etc. that we are planning to store all of our anime on whether temporary or otherwise. While probably the easiest, it is also probably the slowest to use the space on the SD card thanks to low read/write speeds, thus we definitely don't want to use that for storage. Usb sticks would probably fare somewhat poorly as well since their wear and tear is also not the greatest considering we are constantly writing and reading from it (supposedly, though technically it's only really maybe one-two days per week when all the episodes are released). So of course, the best choices would be an external hard drive (spinning or ssd), both of which are decently fast and are pretty cheap now for large sizes.

    With our choice of storage picked, we need to get a type of file system to deal with all of our stuff. This is where things are really user opinion because really it's not like any of the file systems are bad or anything, but we will do a quick rundown anyways.

    -Fat32: Probably one of the few that should be tossed out of the list. Sure you probably won't hit many 4GiB+ files, but you never know when you want that 4K FLAC movie that is like 50GiBs.

    -ExFat: It's kind of inbetween NTFS and Fat32. It allows for large files and has a smaller footprint than NTFS...but it's really only for flash memory, so maybe on an SSD at best.

    -NTFS: The defacto standard for Windows and thus pretty much all hard drives will end up being formatted to it. It's not bad, and if you have a Windows computer, it could be more convenient...though you might need to defrag it eventually...

    -HFS/HFS+/Mac stuff: No.

    -ZFS/BtrFS/XFS/JFS/etc: This is where stuff becomes just a toss-up. If you know the differences between all of them, then chances are you don't need this post to tell you anything.

    -Ext3/4: The defacto linux standard for the time being. Probably a decent pick normally and defragging isn't something you have to worry about, but at this point, you might want to just ditch Windows entirely because while you can use it with Windows, it's not perfect for whatever reasons.

    As an extra note, if you are planning on having multiple hard drives attached or something, it doesn't hurt to look at things like btrfs or xfs or other things. Most of that really comes to preference and what you trust. Since I know and trust ext4, that WOULD be what I would love to format my hard drive to...but I started my collection long ago and thus I've got to stick with NTFS for my current hard drive (converting at this point would take ages with my almost full 2TB hard drive). There is also possibly a larger overhead when using NTFS rather than ext3/4 that could also slow down our limited raspberry pi.

    So we now have a great hard drive to put our stuff in, and hopefully a good file system to go along with it (I won't hate you if you stick with NTFS, but I will hate you if you decide on HFS+ or something). Now we need a way to get a list of the torrents that we want, preferably from an RSS feed because those are still around.


    Raspberry Pi from One

    So we finally have a wonderfully set up raspberry pi that is ready to have a bazillion packages installed to bloat it. First, we need a list of the things we'll need in order to turn this pi into a media server that better be hauling weights for us.
    From start to finish:

  • Some way to manage our raspberry pi if we want to update or something goes wrong
  • A storage place for the stuff we are hoping to use our torrent client for
  • RSS aggregator of some sort to sift through the torrents we are looking for and grab the ones we want
  • A torrent client to actually get them without killing our pi
  • A protocol to serve our media to another device on the network
  • Maybe something to hide ourselves so we aren't blatantly telling the world that anime is all we ever want

  • For the first point, we are going to need to have some way of managing the raspberry pi without needing to actually plug it into a monitor, which takes effort and another keyboard, etc. While you could go with a remote desktop sort of thing, that is really inefficient and wastes resources and so we are going with the best of basics and stick with ssh. Thankfully, openssh is installed by default so there really isn't much we need to do to the raspberry pi except get the IP address somehow. Of course, we could always just use "ifconfig" if we have every cumbersome thing plugged into the raspberry pi, but that doesn't really help much unless you set the raspberry pi to use a static IP. So we are going to set the raspberry pi to use a static IP and make things easier on ourselves. How is this easier? Well that is because otherwise, you would need to be able to find the IP address of your raspberry pi from some other device (which IS possible, but somewhat inconvenient) or have your raspberry pi with a DNS service to resolve a custom domain into your raspberry pi's IP address (also somewhat inconvenient).

    Setting a static IP:
    To set up a static IP on your raspberry pi is actually pretty easy thanks to netctl (remember this is Archlinux, I'm not going to care about other distros). Simply "cd /etc/netctl/examples", check what examples match your setup (ethernet-static, wireless-wpa-static, etc.) and copy that into the parent directory with "sudo cp ethernet-static ..". A little bit of editing is needed, so we will need to use "sudo nano ../ethernet-static" and it should say 
    Description='A basic static ethernet connection'
    Make sure to take note that there is an extra line that says "AutoWired=yes" that needs to be added. You can also change the IP address that is going to be used by changing the line "Address=('192.168.1.XX/24')" to another IP address that you can remember. Also just to make sure, add parenthesis around the "Gateway" value so things don't go awry.

    Almost done, just need to enable the configuration by typing "sudo netctl enable ethernet-static" and your raspberry pi will try to get that IP address automagically after booting. At this point, we can now ssh into our newly static raspberry pi and ditch the cumbersome extra equipment (screen, keyboard, etc.), so feel free to shutdown the raspberry pi and reboot it after placing it somewhere hidden, like under the bed. You can connect to your raspberry pi by using an ssh client (there are quite a few out there) as long as you have your username, password, and the IP address of your raspberry pi. Generally the format is "username@IPaddress" and then after connecting, you are asked if you trust the computer and then you can type in your password to log in.

    From here on, everything that I'm doing is going to be via ssh, not that there is really going to be a difference. Next is to look at the different aggregators that we can use and hopefully pick one to stick with.

    Raspberry Pi from Zero

    I thought I would make a post, or rather a log of setting up my raspberry pi in the event that I have to start over from scratch. Of course, this will mainly be covering the finer details of setting up the raspberry pi, since it's pretty easy to install via NOOBS. While NOOBS is probably easiest, the option is available to install just a certain distribution such as Pidora, Raspbian, etc. if you really aren't planning on using anything extra. In any case, I'm using the NOOBS lite version since I'm planning on having a cable anyways to update the install. With that said, the rest of this post is under the assumption that you have just installed Arch on your raspberry pi and are on the very first boot of your new installation.

    The arch install itself took my raspberry pi around 4-5 minutes and the default login is root/root and probably the first thing to do is to update the system with a simple "pacman -Syu". After updating everything, rebooting is a good idea to make sure everything is reloaded and nothing strange happens.

    Then we set the root password with "passwd" and change the hostname as well to something we could use to differentiate or just for laughs with "nano /etc/hostname". Just remember to add that hostname to the /etc/hosts file (" localhost.localdomain localhost <hostname>).
    Continuing on, we remove /etc/localtime via "rm /etc/localtime" and link to a new one with "ln -s /usr/share/zoneinfo/America/Los_Angeles /etc/localtime". Of course, change the America and Los_Angeles bit to match your local time (to list out all of the options, just "ls /usr/share/zoneinfo/...").

    We could literally call it quits here and just use the root account for everything, but as anyone who knows what they are doing, you NEVER just casually use root for anything. So, we need to create a new user and give him almighty powers that he can use whenever he wants. This can be done with "useradd -m -G wheel -s /bin/bash <username>" and to break the flags down, -m is to make a /home/username directory to go with the new user, -G is the group or groups the user is in, and -s is what shell they will use. Now we need to give the user a password, thus "passwd <username>" to change that to something memorable.

    To give root access to users, we are going to need to grab "sudo" from the repo, so "pacman -S sudo" will install that in a snap. Then we open the permission settings for sudo, we are going to use "EDITOR=nano visudo", though if you like using vi as a text editor, that is fine too. Scroll down to "# %wheel ALL=(ALL) NOPASSWD: ALL" and uncomment it by removing the "# " in front of it. Feel free to read the descriptions for the options in the event that you don't want to have the users in wheel to just have unrestricted root access (ie. they need to use their password, etc.)

    For the sake of clarity, you don't need to use the wheel group, you can also use the sudo group or a totally different group if you want. Just remember to have the user in that group and that the sudo permissions also reflect that.

    Next is to set up the time to sync with the internet time, thus we need to enable ntp with "timedatectl set-ntp true" and things should supposedly automagically work.

    Now we can log out of root and log into our new user account. At this point, we have a fully prepared Archlinux install that is ready to become the testbench of the flood of programs we are going to install to create our media server/aggregator. Since this is a lengthy post as it is, and I would rather re-read this stuff on separate posts, I'm going to do just that.