walls.corpus

By Nathan L. Walls

Articles tagged “development”

Benfits of "throwaway" scripting

Saturday, 20 June, 2015 — development

I like listening to concerts from some of my favorite artists like Mogwai, Explosions in the Sky and Hot Chip. Some artists have a definitive place to go for concert recordings, such as Reflecting in the Chrome for Nine Inch Nails.

For most artists, I end-up visiting YouTube and finding a concert and recently, I’ve found a bunch on YouTube:

While watching on YouTube is great, I would like to listen to these concerts through iTunes or on my phone.

I looked up how I get YouTube video converted to audio and found this Meta Filter thread.

I ended up using the following idea, highlighted in this comment:

1
2
3

youtube-dl UUGB7bYBlq8
ffmpeg -i UUGB7bYBlq8.WHATEVER -vn \
  -acodec copy 'Artist -Title I Want.mp4'

Three keys here:

  1. Get the IDs of the videos I wanted to convert from YouTube. I did this manually
  2. Install youtube-dl, which I did through Homebrew
  3. Install ffmpeg, also through Homebrew

While there are plenty of online or graphical tools one could use to convert YouTube videos to audio, the benefit of a command line tool is that I could then use these tools in a couple of Ruby scripts.

A lot of times, writing code involves writing tests and solving a problem through an application. Theoretically, I could have done that here. But, that felt like overkill because, right now, I have eight or so concert videos.

I wrote two scripts to help me. The first is download.rb:

1
2
3
4
5
6
7
8

#!/usr/bin/env ruby

file_list = "concerts.md"
files = File.readlines(file_list)

files.each do |file|
  `youtube-dl #{file}`
end

In the file, concerts.md is in the same directory and just contains a list of YouTube video ids.

Once these were all downloaded, I needed another script to convert the video files to audio files. I also wanted to name the resulting files. I could do both with a simple data structure. So, I wrote converter.rb.

Neither of these two files is doing anything particularly difficult. I’m just running those command line utilties. But, I’m not having to run them repeatedly. I was able to use ls and Vim to get the file names into converter.rb, then regular expressions to coerce the file listing into a data structure. I filled out the :destination keys manually. That felt like a pretty decent balance of effort to automation.

If I use this file much more, I may improve both of these scripts into something more mature. But, without waiting for that to happen, I was able to take care of some very pragmatic automation right now to save me some tedium.

I’ll take that.

Tool Sharpening: March 11, 2015

Wednesday, 11 March, 2015 — development improvement

I’m presenting Intermediate Git on March 31. It’s a one-day, hands-on workshop to build skills from beginner to intermediate with Git on the command line. Cost: $449. You can register here.

Environment + Process tweaks

I separated “Reading and Learning” posts from “Tool Sharpening” posts, as I mentioned I was going to do. These are separate concerns and they’ll be easier to make both types of post separately. My expectation is I’ll have roughly twice as many Reading and Learning posts as Tool Sharpening posts.

Also, with my new gig, I’m a lot more comfortable working on a Pull Request model. Even if I’m the only person working on a project, it’s still good process reinforcement. I’m also using a rebase model. I’ve never cared for merge commits. Conversely, reducing a feature branch to one or two very descriptive commits feels pretty good. I’ll use small commits in the moment when I’m taking small steps to implement a feature. Once I finish the feature, those small steps lose their utility. They become noise. The Pull Request model lets me strip away that noise and tell the larger story about what the commit is doing and why its there.

I also:

  • Adjusted the TextExpander shortcuts I use to generate much of each podcast line to account for wording changes I’ve made
  • Refactored my TextExpander shortcuts for podcasts to create the Markdown link for podcasts with a predictable URL structure, which thankfully, is most of the ones I listen to
    • A further refactoring to do is to transition to a single TextExpander shortcut that calls a Ruby script with which I could retrieve the show notes page and grab the episode title
    • This would allow me to more quickly scrobble the podcasts I listen to, even though there’s not a direct way to get the information out of Overcast.
  • Created some TextExpander shortcuts for fitness and exercise
  • I added some additional Composure shell commands for Git
    • grbc is git rebase --continue
      • Useful for resolving conflicts on a merge or, more likely with the way I work, a rebase before a merge
    • gmt is git mergetool
      • Get me into merge conflict resolution as quickly as possible
    • fbc is git push origin :$1 && git branch -d $1 and allows me to clean-up a feature branch after I’ve merged it to master
      • E.g. fbc nlw-new-feature
      • fbc stands for feature branch cleanup
    • gsu is git branch --set-upstream-to=origin/$1 $1
      • Useful for when I’m working with another developer on a feature branch I started and pushed, but now need to pull changes the other develoer made on the feature branch

Project work

  • Moved Accomplished over to GitHub and fixed up the initial batch of tests
  • Set-up a new Digital Ocean server ($10 credit for using the link) that this blog will eventually be migrated to. Outside of the initial machine creation and user addition, I’ll be doing as much of the set-up and management that I can through Ansible

For some more background on what’s going on here, see the first tool sharpening post.

Tool Sharpening: Jan 1, 2015

Thursday, 1 January, 2015 — development improvement

For some background on what’s going on here, see the first tool sharpening post

Environment + Process tweaks

I added several zsh aliases for some Git actions that are becoming more common for me, when working on feature branches

1
2
3
4
5

alias gsl='git stash list'
alias gssv='git stash save'
alias gss='git stash show'
alias gssp='git stash show -p'
alias gsp='git stash pop'

These are particularly useful when I’m in the middle of work on a repository and I want to do anything else, like run a git bisect or examine another branch.

Another use case I have is searching, with find or with ag, then opening resulting files in BBEdit.

A typical case looks like this:

1

ag -l foo -print0 | xargs bbedit

In this example, any file in the current directory or below containing the term ‘foo’ will be opened in BBEdit. The find version is very similar for cases when I want to open all files with a name containing ‘foo’.

1

find . -name '2014*' -print | xargs bbedit

Rather than type these commands out every time, I want to type a much shorter command and the search string, like so:

1

bbf foobar

or

1

bbg foobar

It would be easiest for me, by experience, to implement each of these in Ruby or Perl. Instead, I chose to implement these as zsh shell functions. I used Erich Smith’s Composure shell library to build the functions. The resulting code looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# author, about, param, example and group are Composure functions
# ag is the Silver Searcher (brew install ag)
bbf () {
    author 'Nathan L. Walls'
    about 'Function to find files matching a term and open them in BBEdit'
    param '1: Term to search for via `find . -name`'
    example 'bbf 2014*'
    group 'utility'

    find . -name '$*' -print | to_bbedit
}

bbg () {
    author 'Nathan L. Walls'
    about 'Func. to open files containing a term in BBEdit'
    param '1: Term to search for via `ag -l`'
    example 'bbg foo'
    group 'utility'

    ag -l $* -print0 | to_bbedit
}

to_bbedit () {
    author 'Nathan L. Walls'
    about 'Func. to open files in BBEdit through xargs, via a pipe'
    param 'None, implied to come through xargs'
    example 'ls | to_bbedit'
    group 'utility'

    xargs bbedit
}

One other use case I have for this sort of thing doesn’t even require opening files, necessarily, and that’s seeing what files I have that have the word “focus” within them, except spec_helper.rb, in my Ruby projects. This way, I can keep from committing code with focus: true in place and save myself the step of rerunning tests and amending a commit. That looks like this:

1

alias foc='ag -l focus --ignore spec_helper.rb

Or, at least it did until I thought about it a little more and wanted to make it a function as well. Now, it looks like this, instead:

1
2
3
4
5
6
7
8
9

foc () {
    author 'Nathan L. Walls'
    about "Find errant 'focus: true' statements before committing"
    param 'None'
    example 'foc'
    group 'testing'

    ag -l 'focus: true' --ignore spec_helper.rb
}

Why not use an alias here? Partly because I’m enjoying using Composure and partly because I believe I might make further use of this foc () function elsewhere.

There’s still a bit more I could do with these functions, namely adding some error checking, but these will get the job done for me well enough.

I also:

  • Tweaked BBEdit 11 settings to limit the scope of clippings used for given languages, which keeps me from getting cross-language clipping pollution. In my case, there was a large set of PHP clippings coming up when I was looking for Ruby clippings. And now, I’m not
  • Found a way to Save All open documents in BBEdit, Cmd-Opt-S
  • Added TextExpander shortcuts related to writing Git commit messages
  • Updated my work machine to the dotfiles set-up I highlighted in the previous tool-sharpening entry
  • Created a Cider profile to get my Homebrew set-up into source control
  • Updated Git to address a Mac OS X vulnerability

Project work

  • No project work since the last entry

Skill improvements

See above regarding working with Composure to write shell functions. Overall, I’m finding Composure a really nice little framework for iterating over commands quickly and making them understandable later. It was starting to write these commands and then trying to remember Smith’s presentation at Triangle Ruby Brigade back in August 2014 about Composure that led to my coworker Steve finding Smith’s lightning talk.

Reading and Learning: Dec. 30, 2014

Tuesday, 30 December, 2014 — development improvement

For some background on what’s going on here, see the first tool sharpening post

This is a longer post, simply because of how much time has passed since the previous entry, largely due to end-of-year holiday preparation and celebrations. I hope you enjoyed your holidays with family and friends as I did.

I’m also splitting out what I read, watched and listened to from technical tool sharpening. These are different concerns on different schedules and just as I want to sustain and improve both sets of practices, I don’t want them unnecessarily coupled. So, here’s articles, podcasts and presentations with technical pieces to follow separately.

Articles read

Screencasts, podcasts and presentations

Tool Sharpening: Nov. 23, 2014

Sunday, 23 November, 2014 — development improvement

For some background on what’s going on here, see the first tool sharpening post

Environment + Process tweaks

I’ve used thoughtbot’s dotfiles repository for years, but I’ve maintained my own fork on GitHub. I’ve also maintained a private sidecar repository for more sensitive things. But, I’ve been out-of-sync with the head of development. Lately, thoughtbot has released rcm which allows for just such a split-brain dotfile set-up. Rather than fight to update my existing set-up, I’ve opened a new repository and ported my existing dotfiles and customizations to a new sidecar repository that fits with rcm‘s methodology.

This project took about two hours of focused time for the conversion and gave me an opportunity to rediscover some pieces of my dotfiles that I hadn’t quite fully grasped previously. When it came time, rcm worked perfectly about updating symlinks from the configured repositories into my home directory. Neat tool.

I also:

  • Fixed a Tmuxinator profile for a work project
  • Created a BBEdit project for keeping project specific development notes in a more collected fashion

Project work

I picked up work on my Dayplan bin script/Gem, primarily organizing what has to get done in a step-wise fashion.

Skill improvements

  • Reminding myself of new BBEdit 11 shortcuts and previously worked with Emacs shortcuts that are available in BBEdit

Articles read

  • “The Sixth Stage of Grief Is Retro-computing”, by Paul Ford
    • This piece has been recommended by a lot of folks over the last few weeks and I’m adding my recommendation to the pile
  • “Web Applications and Security”, by Alan Cox

  • “The unexpected costs of third-party login”, by Taylor Hughes

    • Just what it says on the tin. There are some substantial tradeoffs to using Google, Twitter or Facebook accounts as login mechanisms that aren’t readily apparent when a product team decides to shortcut account creation. One point I found particularly interesting was this:

      As it turns out, supporting multiple sign-in options adds a tremendous amount of complexity. Instead of an interface where a user can only do one thing — even if it means filling out a tedious form — people have to stop and think about options.

      This leads to another unexpected side-effect: When logging back in, lots of people forget how they signed up for your service in the first place, and they end up trying the wrong method.

      Worse, long-lived mobile sessions exacerbate the issue. When users log in again months later, it’s like rolling the dice.

      This results in support emails from users who are very confused about why all their content is missing after upgrading phones — or users wondering why, when they login on the web site, none of their stuff appears.

  • “Tail Call Optimization in Ruby”, by Nithin Bekal

  • “Rust and Go”, by Adam Jacob

  • “A Five-stage Model Of the Mental Activities Involved in Directed Skill Acquisition”, by Stuart E. Dreyfus and Hubert L. Dreyfus

    • This is the original “Dreyfus Model of Skill Acquisition” paper and very accessible

  • Corey Haines, “Short-lived Branches”

    • Haines explores the constraint of finishing a feature on a branch in a day or deleting it and starting over the next

  • “How to Read an Academic Article”, by Peter Klein

    • With the growth of groups like Papers We Love, it’s helpful to have an idea on how to approach reading academically focused material
    • There’s more info at Papers We Love’s Boston chapter with an upcoming meeting

Screencasts, podcasts and presentations

  • Attended Brandon Mathis’ Triangle Ruby Brigade presentation, “Dive into Ruby”
    • A nifty, REPL-driven question and answer presentation for folks new to Ruby
  • Organized and attended Henry Petroski’s Triangle DevOps presentation, “Success and Failure in Engineering: A Paradoxical Relationship”
    • I’ve been a huge fan of Dr. Petroski and I asked him back in January, via note I sent to his department at Duke University, if he would be willing to speak to the group. He was and the talk he delivered this past week was absolutely everything I hoped it would be
    • The talk was recorded by our hosts at Bronto Software. Give it a look.
  • Listened to Is TDD Dead: Ep. 3: “Feedback and QA”
  • Listened to Is TDD Dead: Ep. 4 + 5: “Costs of Testing” and “Answering Questions”
    • These two episodes finished out the series. It is evident that each of the participants has a great deal of respect for the other two
    • I’m far closer to Martin Fowler and Kent Beck’s positions on the value of testing than I am David Heinemeier Hansson’s
    • Despite being further from DHH’s position, I leave the series with a better understanding of his perspective
  • Listened to Accidental Tech Podcast Ep. 88: “Standing on Opposite Sides of the Gym”
  • Listened to Accidental Tech Podcast Ep. 89: “DeLorean + McLaren”
  • Listened to Giant Robots Smashing Into Other Giant Robots Ep. 121: “Prolificness (Allison House)”
    • Something House mentioned during the podcast was the notion of a second shift of personal projects, e.g. a 20-minute portrait painting where the painter has to focus on what’s critical
    • The key, it sounds like of second shift personal work is making it consistent and sustainable
    • This is something I aspire to, but still find myself struggling with
  • Listened to Bikeshed: Ep. 1: “Sandi & Derek’s Rules”
  • Listened to Back to Work Ep. 193: “Disappointment Delivery Mechanism”
  • Listened to Back to Work Ep. 194: “The Company of a Clown” Listened to Ruby Rogues Ep. 179: “Accountability and Diversity with Meagan Waller”
  • Listened to Ruby Rogues Ep. 180: “Barriers to New Developers with Kinsey Ann Durham”
    • I can’t think of a bad episode of Ruby Rogues, but these two are exceptionally good ones – windows into experiences I can’t have (being a female developer) or haven’t had in a long time (being a new developer). I highly recommend giving these both a thorough listen

Programming note

First, I can explicitly state the next entry will be in December, likely Dec. 7. Second, I’m entertaining the idea of breaking this larger post into smaller components. Once a week for all of this feels a little thin and two weeks starts to feel a little much. I’m thinking the articles and podcast/presentation sections are both getting a little long and may benefit from separate publication weekly with my project work, environment tweaks and skill improvements happening more on a biweekly basis. I’m not set on that, but I think I’m going to experiment with it.