Categories
Hardware Tools

Upgrade Your Storage Every Decade!

I got a home NAS (Network Attached Storage) with a formatted volume size of about 11TB (4x4TB drives in a custom RAID). I've had a few local USB/Firewire-based home storage solutions in the past such as a Drobo, but this time I wanted to go to the network because I thought it would be more flexible. The drivers were:

  • Its good to upgrade storage periodically over your lifetime (see below)
  • The NAS is more flexible that a single-computer USB-based connection
  • It has more storage
  • The NAS device has a lot more features than a simple storage array

More on all of this below.

The model I chose is a Synology DS920+.

Synology 920+ NAS
Synology 920+ NAS

The DS920+ has four drive bays and two m.2 slots for a read/write SSD cache. This is a nice feature because you can get inexpensive high-capacity spinning disk drives and use the SSD as read/write cache to mitigate the performance issues you'd normally see on low-rpm spindle-based storage such as horrible parallel reads/writes and slow random access/seek speeds.

I got four Seagate IronWolf 4TB NAS Internal Hard Drive HDD.  These have a modest 64MB cache and only spin at 5900rpm. I also filled both m.2 slots with a pair of Samsung 500GB  970 EVO SSD's.

The DS920+ also offered a 4GB RAM expansion, bringing total RAM to 8GB. This is useful since the NAS is really a Linux computer and my gut says 8GB will give the OS more breathing room for disk cache and running various apps and protocol servers it supports.

On that note, the Synology DS920+ has a ton of apps and services and I've only scratched the surface on what I use. Some of the highlights:

  • SMB server (for WIndows sharing)
  • AFP/Bonjour server (for Mac)
  • rsync
  • NFS

SMB and AFP are pretty slow protocols and I always hate using them for large file transfers, like the multi-terabyte transfers I need to make to get all of my home videos and photos off my old Drobo. I found a great writeup on the performance of these protocols by Nasim Mansurov, here at the Photography Life blog. These protocols are great for general use, but not for the initial data-loading phase.

Part of my apprehension is not knowing the state of a large transfer, particularly if its interrupted. If I moved files and the transfer was interrupted, were they really moved?  Sometimes I'm dealing with 50k files and it's not easy to get a warm and fuzzy feeling about whether a large file transfer worked or not even if it appeared to finish. Sure, when the copy is done I could compare file counts and byte sizes between both source and destination. This would give me some confidence that I can now delete the source directory, but that's not the real problem.

The real problem is managing a large transfer and being able to optimally start/stop/resume it at will. This is trash using a GUI and AFP/SMB. For instance, if I want to resume a copy, do I need to copy the whole thing again? Do I manage the copy by doing one folder at a time and sit and wait for it to finish before starting the next folder? LOL, I've been there! Also, what happens when I find destination files that already exist? Walking into my office at 7am to check on the progress of a a twelve hour file transfer only to find that twenty minutes after I went to bed the system politely prompted me about how I want to handle a destination file conflict. I never want to be that angry holding a cup of hot coffee ever again. Screw all that.

The answer of course is rsync, a tried and true utility that's a foundation of Linux. Since the NAS is a Linux machine, it's got an rsync facility. rsync is single-purposed yet sophisticated piece of software that runs on both ends of a file transfer. The client and server both have access to their respective file systems and can negotiate a large transfer efficiently. If it's good enough for Fortune 500 companies to use in production, it's good enough for home videos of my kids fighting in their diapers.

rsync negotiates file listings and checksums on both sides of the transfer and will always do the right thing in terms of resuming a partial transfer. It's like magic.

To get this to work smoothly, I had to root into the NAS drive and change a few SSH options to allow for a more or less automated transfer. Right now I'm in the middle of what will be a 2+ day transfer of 1.6TB of home videos.

It's Good to Upgrade Storage

On a final note, I wanted to say one simple thing about why I like to upgrade my storage every decade or so. It's based on a few simple points:

  • Hard drives have a life expectancy and they don't last forever
  • New connectors and protocols are constantly getting faster and more robust
  • Old connectors and protocols are always becoming obsolete. If you plan the frequency of your copies properly, you'll always be in an era where both the old and new technology are still around and you can get adapters and whatnot to ensure the transfer is possible
  • I like to play with new things

Ok, the last point is probably the real reason. You figured me out.

Categories
Productivity Tools Writing

On Beta Reading

I've been on both sides of the beta reading spectrum in my journey to write my first novel, The Harvester. Rewarding is the best word I can think of right now when I think of both the experience in beta reading for others as well as the experience in working with people who've read my manuscript. My friends have been wonderful and this past January I joined a writing group and the past six months has been a whirlwind of discovery.

That said, know that whatever I say here are the words of a beginner. I know nothing. I am learning as I go.

Categories
Tools Writing

Ulysses for Writing Prose on Mac

I use Ulysses app for Mac OS X for all of my writing. When you write in Ulysses you're working with plain text and you perform basic formatting with markdown syntax. What I love about Ulysses is how simple it is. Yet the application's simplicity does not mean it's trivial. In fact, it's extremely powerful. I am writing this post to talk about how I use it to write.

Categories
Tools

Generate PDF’s and ePub with wkhtmltopdf and Calibre

In a previous post, I wrote about how I use GNU make to manage dependencies and generate html files from markdown source. In this post, I'll build on that and use the html to generate PDF's and ePub files.

MultiMarkdown can generate PDF files using LaTeX. For some reason, I never got that to work. I tried on multiple Macs with clean installs of MultiMarkdown and a variety of LaTeX apps like mmd2tex, the LaTeX support files for MultiMarkdown, etc. I failed each time. Plus, I don't really want to re-learn LaTeX. And I hate typing in LaTeX.

I know HTML. I know CSS. So I sought out tools to help me leverage those skills.

Leveraging HTML

It's obvious I didn't want to load the HTML into the browser and use the OS to Save As PDF. That would be lame. I wanted to generate this stuff at the command-line, and in scripts, and automate things. I also had an intuition that Webkit would be exposed in more ways that simply embedded into Chrome and Safari. So, I searched for "webkit pdf." What's the first link I found?

http://code.google.com/p/wkhtmltopdf/

Jackpot! This is an awesome set of commands that allow you to feed it an HTML input and have it generate a crisp PDF. Click on the HTML and resulting PDF:

http://primordia.com/upload/lorem_ipsum.html
http://primordia.com/upload/lorem_ipsum.pdf

The command-line to generate this is as follows:

[code wraplines="true"]
wkhtmltopdf --page-width 5.5in --page-height 8.5in --margin-top 0.25in
--margin-bottom 0.25in --margin-left 0.25in --margin-right 0.25in
--load-error-handling ignore lorem_ipsum.html lorem_ipsum.pdf
[/code]

QED.

ePub

Generating an ePub format was also a goal. I googled all over and found a bunch of tools. The one that looked nicest was Calibre. Calibre is a complete e-book management app. It has crazy features:

  • Understand a gazillion e-book formats
  • Can import pdf books, html books, etc.
  • Can edit e-book meta-data
  • Add cover graphics
  • and lots more

But I  just wanted an ePub file. I first tried to see if I could generate an ePub myself. The format is a zip file with HTML inside and a bunch of crazy metadata to create table of contents, chapters, etc. I didn't want to do all of that work so I'm glad I found Calibre. But like pdf's, I didn't want to load the GUI and manually generate ePub files. So I inspected the .app package and inside I found SOLID GOLD. There are a bunch of command-line apps that the GUI uses to do all of the work. Now that's a programmer who knows what he or she is doing! I was able to add the ebook-convert program to my path and invoke it as follows:

[code wraplines="true"]

ebook-convert --no-default-epub-cover --base-font-size 12 --keep-ligatures
--margin-top 10.0 --margin-bottom 10 tmp.html tmp.epub

[/code]

Now I have lorem_ipsum.epub!

I like to use Adobe Digital Editions e-reader on the Mac because I don't need to add an ePub file to any kind of Library. I just open an ePub file and a viewer displays it. No "import" process which is stupid since my ePub files will change so often as I write. You can of course use Calibre to view ePub files, Kindle, and a zillion other apps to do the same thing (albeit with the extra import step).

Width Problems

But there is a very bad problem. The formatting is horrendous! Check this out:

Cut off

The whole thing is cut off at the right? Why? Because my HTML specifically set a width so it would be small like a real book. This was not really necessary, but makes the HTML easy to read when your browser is maximized. If I hadn't done this, text would wrap to 100% of the width of the browser and lines would be too long to read comfortably. But e-readers don't want you to specify widths. Users have lots of different devices. Users play with font sizes and orientations and nothing can be easily predicted. So, you want an HTML file that doesn't specify any width, like this:

http://primordia.com/upload/tmp.html

The offending code was in the CSS:

[code wraplines="true"]

body {
width: 6in;
margin-left: auto;
margin-right: auto;
}

[/code]

So I just removed the width setting. Unfortunately, either calibre or wkhtmltopdf doesn't respect multiple STYLE tags so I could not simply override the width when generating ePub with a second stylesheet (the way CSS was designed!). I guess I should file a bug report. Anyways, I punted and just cloned the CSS and use the ePub version of the CSS when I want to generate ePub. This is lame but cpress handles it. At some point, I'll create a facility in cpress to merge css streams so I can have one master CSS and an ePub version which simply gets rid of the width. For now, two CSS files. Here is an image of the resulting ePub as viewed on my iPhone in iBooks Reader:

lorem2

How fucking beautiful is that?!

I think that's all I'll say for now. I have lots more to share with you. In my next post, I'll talk about how I aggregate multiple markdown files into as single markdown file, how a table-of-contents gets auto-generated, and how I script the generation of my  html, pdf, and ePub artifacts with crontab and sync everything with Dropbox.

Categories
Tools

VIM spf13 distro

I am now a VIM spf13 user. That is all. Carry on.