8/17/14

Android and LibMTP

So obviously being a cool guy, you have an android device or two that you may or may not want to connect via usb to your computer. While in windows, mtp is either barely passable or you just can't get the shit to work at all. In both cases, mtp is annoying to use and all around terrible in comparison to the old way of just mounting directly in my opinion. Fortunately, the Archlinux wiki has an article on that (as with almost everything else ._.) and all that really needs to be installed are gvfs and libmtp, though apparently gvfs-mtp is still available as a separate package. It has already been merged upstream and released with gvfs 1.15.2, so any version after, you probably don't need gvfs-mtp.

As a quick note, installing just gvfs and libmtp on a vanilla Arch install as of right now won't give you a working auto-magically mounting android MTP device. This is because gvfs apparently also needs a polkit authentication agent like pkttyagent, lxpolkit, mate-polkit, polkit-gnome, or polkit-kde...and that needs the session to be started with dbus-launch. Pkttyagent is included by default as a console auth agent, though I use lxpolkit anyway since having a GUI agent allows me to open things like GParted with pkexec.

Usually this is enough to get things working, but sometimes the device might have not been added yet to their massive list of udev rules. Of course instructions are on the wiki so you can just add those, reload udev rules, and maybe reboot.

Now this still has the same problems that mtp does on Windows, primarily that what is shown on the computer and the phone do not always show the same things after manipulating files with the phone because that is how MTP supposedly works. Anyway, with that MTP should supposedly be working and all sorts of magic happening with any file manager that supports gvfs (PcManFM, Thunar, etc.).

Raspberry Pi from Five

So we are finally at the end of the great raspberry pi journey for media serving and the last thing we really need is a way to protect our precious collection. This can be encompassed into two main ideas: protecting what goes in and out of your computer via the network, and protecting the stuff on your filesystem. Most of this I can't go in-depth about, but the general idea is to have a firewall for filtering what stuff comes in, encrypt and proxy around stuff that goes out, and encrypt the files in your filesystem. I have next to no opinions for any of these since I do none of them :D. But I'll still list stuff off the top of my head that pertain to it.

Firewall:
- IPTables: Pretty much the baseline for firewalls on Linux, most of everything else is a frontend.
- Any of the IPTables front-ends (Gufw is apparently one of the popular ones)

Encrypting Traffic out:
- Https for everything (both firefox and chromium have plugins for HTTPS everywhere)
- SSL/TLS for everything (this and HTTPS are more about protecting your web browsing)
- DNSCrypt: Helps prevent spying and man-in-the-middle attacks because apparently your IP can still be leaked to the ISP even with https/vpn/ssl, and guess what, it just wraps your DNS conversation stuff in SSL as well :D.
- A VPN service...which costs money...Pretty much, they will encrypt all of your data that you send out and bring in. Apparently PrivateInternetAccess is one of the better ones that also allows torrent traffic.
- A seed box: It downloads for you and then you can get the files over ftp or something...aka connect your raspberry pi to a network that isn't yours and use ngrok to be able to connect to it...but you better be damn confident that it's not going to go down easy.

Encrypting File System:
- Once upon a time it was TrueCrypt...maybe there will be a derivative somewhere later on.
- Tomb: Not only is it witty, but it's quite good...as long as you don't lose your key and password...which you still need to hide somewhere.

Some extra tidbits: You may have noticed that for a "hide your shit" post, there isn't any mention of the tor project and there is a simple reason for this. You should never use bittorrent over tor....ever....because it wasn't made for bittorrent traffic, it just won't work and if you do it, your results WILL be subpar. Also, sending your traffic through these things will show a performance hit in torrenting speed, but at least you can't get rekt until they have a probable cause that you have amassed a huge amount of something and find your tomb on your confiscated computer and forcefully find a way to get the key out from your cold, dead hands. Unfortunately, this means that you need to manually handle your tomb file and optionally make a script to clean up your dirty work so only you know where it is.

Raspberry Pi from Four

So we are finally complete with all of our wondrous automation on our raspberry pi and thus, now is the time for a recap of what remaining stuff we need to deal with.


  • A protocol to serve our media to another device on the network
  • Maybe something to hide ourselves so we aren't blatantly telling the world that anime is all we ever want

  • Today is a day where we will talk about another subject that I know next to nothing about, but that is totally fine because the internet never lies. Only truths here... Anyways, we are just taking a quick poke at a couple of major types of file transfer protocols so we can stream stuff...maybe...Mostly, it's a preference thing and word from other people about the general performance differences between the different protocols, so I can't hate you for any choice you decide to go with. Of course with that said, it's probably best to get the lightest and fastest because the raspberry pi has a single controller for the ethernet and usb ports, thus it's already shoveling stuff through a pipe as it is. 

    • FTP: It's better than SMB at least. Worst case, it's always worth it to give ftp a shot if all else fails.
    • FTPS: The more preferred way of ftp usage because we need to be secure about our anime. 
    • SMB: Normally a pick if you have a windows computer, but you don't because I say so. Also, it's not the fastest out there, therefore SMB isn't needed.
    • SSHFS: This should really only be compared to NFS, and unfortunately NFS wins in the speed department.
    • SFTP: Kind of follows suit with SSHFS, though it's not really surprising since it is also on top of SSH.
    • SSH: A really clunky way to go about streaming (ssh user@ip "cat path/to/video" | mpv), but it's available if shit really hits the fan or you just don't want to use anything else.
    • UPnP/DLNA: Literally only for streaming media and nothing else. No file manipulation or anything, so you would have to ssh into the pi to manage stuff. Not particularly bad, but being able to manage would be nice.
    • RSync: Not really streaming so much as just copying it to your client computer, but I guess if you sent it to /tmp/ or something it could be considered streaming.
    • NFS: Probably the best choice out of the group in terms of flexibility and performance, and the overhead isn't all that bad either.
    Now, I haven't done any testing on the performance of ANY of these really. At best, I have tried nfs, ftp, and UPnP/DLNA, and in the end I preferred the nfs route. Thus, we are going to set up an nfs server.

    First things first is to grab the nfs-utils package for both the server and client and make a new folder to hold all of our stuff that we are going to serve out. A simple "sudo mkdir -p /srv/nfs4/foldername" would do just fine unless you plan to split up your stuff and mount each to separate folders (ex. Music, Anime, Manga, etc.). Then we need to bind the mount point of our hard drive to the serving folder, this can be done manually with "sudo mount --bind /harddrive/path /srv/nfs4/foldername" or by adding another entry to the fstab to mount for you at boot that follows the lines of "/harddrive/path /srv/nfs4/foldername none bind 0 0". Then we need to set up what to export aka serve.

    Opening the /etc/exports file, we want to write down what we want to serve, who we want to serve, etc. This should be pretty easy to do (even has examples) so something along the lines of "/srv/nfs4/foldername  192.168.1.0/24(rw,async,no_subtree_check,nohide)" will work just fine. Supposedly async and no_subtree_check will make the transfers just a tad faster, which is always nice. After that, "exportfs -rav" should update the exports and you should be ready to start the services with systemd. I would suggest enabling the services and then rebooting rather than just starting, but either is fine. Just "sudo systemctl enable nfs-server" should start up everything needed on the server.

    On our client linux box, we need to set up our nfs client which can be done by enabling rpcbind.service and nfs-client.target. To check our setup, you can use "showmount -e" or "showmount -e raspberrypiIP" depending on if you are on the server or client respectively and to mount the nfs share client-side is a simple "sudo mount rPiIP:/srv/nfs4/sda1 /path/to/mount/point".

    Supposing everything is working, we can optionally continue on and discuss about mounting automatically using fstab, systemd, autofs, etc. In my opinion, the "Tip and Tricks" section of the NFS page in the Arch wiki has a rather nice "Automatic mount handling" section that would probably work well in the event that our raspberry pi goes down for some reason (which might happen).

    Once mounted, you should be able to read/write to it per usual and magical greatness should ensue. Some things to note:
    - You can tune the performance by modifying the rsize and wsize options of the client-side mount.
    - Streaming from NFS works well with animated shorts, but the same can't be unanimously said for large files. It may take a short amount of time to cache some of the video before you can have a smooth playing experience. Of course, this can be somewhat mitigated by tuning the performance.

    7/20/14

    Raspberry Pi from Three and Three Fourths

    Finally we are at the end of setting up our magical torrenting client and all we need to do is have a way to add our anime to an rss watch list. Luckily enough, we have just the plugin that can deal with RSS feeds, but to keep things in order, first we need to decide on a feed to follow.

    The normal sites that I assume are standard when it comes to searching for this kind of media: Nyaa.se, Tokyo Toshokan, BakaBT, and ShanaProject. Out of these, ShanaProject is probably the most anime-oriented and is kind of specifically a site just for tracking anime and has user specific feeds that you can use to follow and track different fansub groups for different anime. It's the default feed of choice for me, but there are perks to using the others as well. Something to note though, Shanaproject along with Tokyo Toshokan pretty much aggregate torrents from Nyaa (at least in terms of anime), so there isn't much point to using all three of them.

    To give a quick run-down of the options aside from ShanaProject:

    • BakaBT: A "community" that dislikes leechers and doesn't seem to have any RSS feeds, making automation a bit of a bitch.
    • Nyaa: The defacto tracker for pretty much all of the fansub groups. If they aren't on Nyaa, they are just retarded somehow.
    • Tokyo Toshokan: The best option for all types of media from Japan. It aggregates anime, manga, music, etc. and has a customizable RSS to boot. Also, it has some re-encoders (hi10ani.me) if they actually put up torrents.
    Anyways, after picking a feed to follow, we just click on the nice little rss icon in rutorrent, add our first rss feed and give it a name. Clicking on the rss icon afterwards now will bring up the RSS manager where all things great and automated occur. It's pretty simple from there, just come up with a name for your new subscription (usually I just title it the name of the anime), a regexp pattern to look for (ex. "/Akame ga Kill.*\[FFF"), the feed to watch, and a subfolder to download it into if you want (it defaults to the downloads directory of rtorrent). After that, the ratio rule and group from the previous post will deal with removal.

    At this point, if you added the anime for each running season, your setup would automatically download, seed, and remove your torrents for the entire season without you needing to do a thing. It is probably possible to add an extra script to automatically generate even the configurations for the rss, but I currently don't have a script for that (in due time, I will get that since there happens to be a rather nifty link available on MAL).

    To take rutorrent a little bit farther and make it actually decent as a torrent client for things like older anime series or episodes that are not on the latest feeds of Nyaa or Tokyo Toshokan, I made some extra search engines for the extsearch plugin (thanks to the wonderful and fairly simple examples of the other engines). Just save them into /srv/http/rutorrent/plugin/extsearch/engines/<searchengine>.php files and they should magically appear and be ready for use.

    Nyaa: http://pastebin.com/D5S6NCyi
    SukebeiNyaa: http://pastebin.com/fLgXAAxu
    TokyoTosho: http://pastebin.com/mS67ZqnX

    Anyways, with that we FINALLY have all of our torrent client/rss aggregator woes fleshed out and out of the way, but we still have one little extra that we can do. After setting up auto ssh authentication (this can be found anywhere on google) from the raspberry pi to your obviously linux-running computer, you can make a rtorrent_notify.sh script to send a notification or something to your computer for when a torrent completes. My script is really simple:
    ssh user@host 'DISPLAY=:0 notify-send "New episode ready!" "$1"'  

    -Extra Tidbit-
    There is another website that could be used for tracking anime releases called anitor.net but that website really isn't useful for automation either (so it's kind of like BakaBT without the community and other useless stuff). It's pretty much a site that shows the latest releases in a nice list and has some fancy scripting that is nice if you are doing things manually.

    7/18/14

    Raspberry Pi from Three and a Half

    So with our rutorrent and rtorrent set up, we can now move on to tackle more first world problems and start this stuff on boot. The main goal is to get lighttpd and rtorrent up and running when the pi starts up, so that we can instantly go to rutorrent after it finishes booting, and then do a little bit of configuration to rtorrent and rutorrent to smooth out some of the edges.

    First is to install tmux or screen, though which one in particular is a toss-up (and I prefer tmux). After that, we need a systemd service to start tmux/screen and rtorrent, as well as stop it. Of course the arch wiki has a nice entry that you can pretty much copy/paste, though I like keeping the rtorrent as the full name for things rather than rt. In any case, "sudo systemctl enable rtorrent@user.service" and similarly for lighttpd.service will start both at boot so you don't have to do that manually and you can check your rtorrent by ssh'ing in and opening that tmux/screen session.

    We are going to be taking a look at the .rtorrent.rc file to add a couple of extras that we'll need and other things. As a warning, pretty much all of the configurations in the .rtorrent.rc are also available in the Rutorrent webui, but for whatever reasons (maybe some permissions problems somewhere) it doesn't seem to save those configurations across reboots. So things like the download directory (best to set to the highest folder in the hierarchy of your anime), session directory, port range, port random, encryption, etc. should be set using the .rtorrent.rc file (and you should set them). At the very end of this file, we are also adding two lines:

    execute = {sh,-c,/usr/bin/php /srv/http/rutorrent/php/initplugins.php &}
    system.method.set_key = event.download.finished,notify_me,"execute=~/.cache/rtorrent/rtorrent_notify.sh,$d.get_name="

    The first line is to load the plugins for rutorrent when rtorrent starts up so our feeds start up. The second line on the other hand is COMPLETELY OPTIONAL and is pretty much just executing a little script called "rtorrent_notify.sh" after a torrent finishes downloading. Supposedly with a little magic and some bash, you can send a great little notification to your computer from your raspberry pi that tells you when a new episode is ready.

    The rest of the configuration can all be done with rutorrent and some time should really be put to exploring all of the settings available, but there are some quick things that I will highlight:
    • General>User Interface>Theme (the oblivion or dark themes look pretty nice)
    • Ratio Groups>Any number
    The first bullet is just a preference thing, but the second is half of the staple way of automatically removing torrents from rtorrent when it's finished downloading. It's fairly easy to understand, just name the ratio to any name, choose a minimum percentage that you want to seed back to your peers, maximum doesn't really matter, a minimum amount in MBs that you want to seed back, and for how long (or just ignore the time all together), and then set the action to remove. If you wanted to be a real asshole, you could set all of it to zero and then as soon as your file finishes downloading, it gets removed or you could be a decent person and seed until you have given as much as you have received. To complete this automatic removal, we need to make sure that every torrent gets set to this ratio, and we can do that with the Rules Manager (button right next to the help, second option).

    Pretty much just make a new rule where "if the torrent label contains <blank>", set the ratio to your new ratio. And by blank, I literally mean just a blank entry. With that, almost everything is taken care of in terms of our torrent client and we can look at the way we are going to be automagically adding new torrents to our heavily under-powered torrent box.

    7/17/14

    Raspberry Pi from Three and a Fourth

    So getting rutorrent onto a raspberry pi is not nearly as amazingly easy as deluge, but hopefully it will be better in the end. Since a lot of the typing work has already been done, I'll just link to the arch wiki which should get you most of the way there. Just make sure to copy the template .rtorrent.rc to your home directory with "cp /usr/share/doc/rtorrent/rtorrent.rc ~/.rtorrent.rc".

    After following the wiki, it brings you to the point where things almost fully work, but the log of rutorrent (when you open it in a web browser) gives you some of the issues it encountered.

    The first of these might be something along the lines of the rtorrent user needing to have read/write permission to the tmp directory. Now I'm no expert in dealing with permissions, but to make things simple and quick to fix, just make a new directory in the /tmp folder with "sudo mkdir /tmp/rt" and then change it's permissions with "sudo chmod 777 /tmp/rt". Finally, taking a quick visit to /srv/http/rutorrent/conf/config.php, at the very end there is a line that says "$tempDirectory = null;" which you change to "$tempDirectory = '/tmp/rt';". Instantly problem solved.

    The other issues that probably got logged were about the plugins. Some of these plugins you might want to disable (like the rutracker_check one), and if you want the other plugins (mediainfo, screenshots, unpack plugin), just install the packages that they are asking for (mediainfo, ffmpeg, unzip, and unrar respectively). The mediainfo, unzip, and unrar packages are pretty small in comparison to ffmpeg, and have their uses in other places as well...ffmpeg on the other hand is about 120MBs, but I'm still going to probably install it anyways. Now is also a good time to take a look through all of the plugins available in rutorrent (all the non-3rd-party ones anyways) and disable unneeded ones. It's not a huge hit in performance leaving them on, but meh, I'll take anything to lighten the load.

    Now these problems are only what rutorrent can show you. There are more problems hidden in the /var/log/lighttpd/error.log that you can also spend time fixing :D. Most of the fixing is going to be in the /etc/php/php.ini, so open that up somehow and we want to add some extra directories to that open_basedir line after "/usr/bin/" to get our disk space plugin to work. Just change the end to "...:/usr/bin/:/:/usr/local/bin/:/usr/local/sbin/" but don't close the config file just yet.

    Next is to fix the issue that get's spammed into the lighttpd error log about the time not being set. If you are using nano, use ctrl+w to open the search function and search for "date.timezone". This will bring you to the setting in the config where you want to uncomment the ";date.timezone =" and add your timezone to the end. In my case, I change it to "date.timezone = America/Los_Angeles" and the value for you is just the same as when you set up your timezone on the raspberry pi. After that, you can save the config, kill any php-cgi that might be running, maybe reload the systemctl daemon, restart lighttpd and give your rutorrent install another check.

    Finally, we can clean up the horrid mess we made of everything and make things a bit nicer with starting at boot.

    7/16/14

    Raspberry Pi from Three

    So with a great file system on your hard drive, we need to do a quick addition to our /etc/fstab to mount it at boot, but first is a recap of the other stuff we still need to do.



  • RSS aggregator of some sort to sift through the torrents we are looking for and grab the ones we want
  • A torrent client to actually get them without killing our pi
  • A protocol to serve our media to another device on the network
  • Maybe something to hide ourselves so we aren't blatantly telling the world that anime is all we ever want

  • Now to mount our drive at boot. Just pop open /etc/fstab with something like "sudo nano /etc/fstab" or any other preferred text editor. You will need to know where your block device is (/dev/sda1, /dev/sda2, /dev/sdb1, etc.) and just slap an entry along the lines of 
    /dev/sda1     <some mount point>   <file system type (ie. ext4, ntfs, )>         defaults       0     0
    If your device mounts and the permissions are wrong, you might not be able to write anything unless you use root. When this happens, add "defaults,uid=1000,gid=1000" or whatever options you are mounting your drive with. And with that, we can finally continue towards tackling the next major issue on our list of things to deal with, the RSS aggregator.

    There is a small number of programs at our disposal that we can use to deal with this problem, with the most notable ones being sickbeard (not really) and flexget. After that there are some others like rssdler, couchpotato, etc. that I didn't look into. Aside from these, there are also some RSS aggregators that are included (usually as a plugin) in torrent clients like deluge or rutorrent (frontend to rtorrent).

    Sickbeard's support for anime is shoddy at best, with the main way being a branch of sickbeard that isn't really maintained, making it kind of pointless...and while flexget is better, I just never liked flexget and so I didn't try it. Rssdler might have been a good choice, since it looked pretty lightweight to match rtorrent, but after 5 years of next to zero development activity, it's pretty much a dead project.

    Thus, I really looked towards deluge and rtorrent as being the grand solution. Setting up and preparing deluge is amazingly easy and primarily relies on a plugin called YaRSS2. Simply install deluge, start the daemon, connect with a deluge client, and install the plugin. After that, you are just about ready to start reading rss feeds and auto-downloading the torrents. Since the wiki is probably loads better than any explanation I could pull off (it has examples), the link is here. Rtorrent on the other hand...has you going through a couple of hoops, but in my opinion, those hoops are just barely justified.

    When using deluge, you would need to use a client and while there is a plugin for a webui, it unfortunately doesn't have a webui for YaRSS2 which is our primary way of automating our anime downloads, meaning we would either need to utilize a script which would add a new entry into the YaRSS2 config after we type in the anime title somewhere or use the gtk client. At the point where you are going to just use the gtk client to connect to the deluge server, you might as well just set up the server on your computer and use deluge in classic mode. Rutorrent on the other hand has an RSS manager plugin and of course, since the whole frontend is a webui, the plugin also has a web interface. This means that when you check your Raspberry pi's torrents, you can manage every torrent-related thing from a single webpage without needing a client (managing it on-the-go).

    Since setting up rtorrent+rutorrent is a bit of a journey, that will have to be in the next post.