ElectroBus

Random musings ッ

What Do I Want to Be When I Grow Up?

| Comments

I sort of fell into IT a long time ago. I was always good with computers and one opportunity led to another. I’ve bounced around in roles quite a bit, gaining valuable experience, but frequently not loving what I do. I always seem to look to the next opportunity.

My favorite position was building a data warehouse, which lasted for a couple of years. But eventually I just ran out of enhancements to do. The company had all of the data sources working, a reliable ETL setup, and several analysts comfortable working with the toolset. Looking back, perhaps I should have left that company then, taking those fresh and marketable skills into a new consultant path.

Instead I did the safe thing — stuck with the company and migrated through a few new roles, implementing VMware, SAN, doing lots of security training, running IT from a technical level, “promoted” to management. Taking this path led to a lot of additional skills and experience but, alas, true career happiness has been elusive.

I finally figured out that management is something that I am capable of, but don’t love. My next job was as a system engineer, which was great until I inevitably got promoted to management.

So I’ve been casting around, trying to find my true “passion” to pursue. I read Chad Fowlers excellent book “The Passionate Programmer”, which has some great advice on building career capital and really launching yourself into the community as a leader.

I’ve always enjoyed software development, so next I picked up Bruce Tate’s book “Seven Languages in Seven Weeks”, and through some kind of magnetic attraction went right to the Erlang section. Erlang has some really cool features, but the coolest may be “hot” code loading — upgrading modules on a running system with (hopefully) zero downtime. Anyone who has been up late at night deploying a EAR file to cluster of JavaEE servers with the requisite downtime can understand the excitement at this notion. A passion found?

One thing I picked up from Chad’s book is to pursue something important, but not yet commoditized. For example, India has a huge number of Java developers. Unless you are a world-renowned expert, Java development is unlikely to lead to more than a mediocre salary and a succession of jobs that eventually get outsourced. Erlang doesn’t appear to (yet) be a target for this mass commoditization. So I pursued it.

Knowing nothing of Erlang, I worked through the Seven Languages chapter on it, then got “Learn You Some Erlang for Great Good!”. This started out well, and there was an Erlang Factory Lite nearby, so I went to that too. In my goodie bag was a copy of “Introducing Erlang: Getting Started in Functional Programming”. I then found that there was a companion book to this “Études for Erlang”, which can be read online for free.

By this time I was getting bogged down in learning Erlang, and realized the massive conceit on my part that I could read a couple of books, do some programming exercises, and jump into some kind of important role in the Erlang community — maybe contributing to one of the many excellent open source projects.

I had also found a number of great podcasts, among them Mostly Erlang and (more importantly) Ruby Rogues. Both of these podcasts feature “picks” and the end of the show where the panelists give links to new / useful / exciting / fun / whatever stuff they think is interesting. One of these was from Episode 136 where James suggested So Good They Can’t Ignore You: Why Skills Trump Passion in the Quest for Work You Love

I’ve been reading this book with great interest, as it dovetails nicely with “The Passionate Programmer”. The trick is to not find a passion and then follow it, but to gain skills and experience through deliberate practice. This gives you valuable skills that you can then trade on for money, position, power, etc. Part of this is very discouraging — deliberate practice is hard work, but it is also nice to see this laid out in an easy to consume manner. Also discouraging is the realization that I simply don’t have a world-class understanding of any one system. However this was tempered with the realization that I have deep experience in a wide range of systems over the past 20+ years, so it’s not as if I am starting from zero.

So what’s next for me? Is Erlang development my true passion? It is literally too early to say. I need to develop my own deliberate practice strategy. Given my past love of data warehousing / ETL type work perhaps I should focus on a “data layer” system. NoSQL is a big thing at the moment, so perhaps a deep-dive into Riak or CouchDB would be a way to combine both Erlang and Database interests.

Leaving AWS’s Free Tier

| Comments

It’s been a good year on the AWS free tier, but the “free” is over. It is hard to tell how much my instance will cost to run, but $10-20 a month looks likely, and for that kind of money there are a lot of other options.

To date I’ve been running:

  • A copy of tt-rss to replace Google Reader
  • Wordpress
  • A spot to temporarily stash files
  • An ssh proxy server to provide secure Internet access

tt-rss

I recently switched from tt-rss to BazQux, and that’s been great. BazQux is supported in my favorite iPad reader, Feeddler, and has a nice web interface too. Best of all, I just pay the annual fee and don’t have to deal with it otherwise.

Wordpress

Hosting my own Wordpress was overkill. I see little point in running a dynamic database-backed application for my simple blog, and decided to go with a more basic static site generated from templates and published up. I played with VoodooPad on my mac, and it has a nice “static” publishing feature, but then I found Octopress and haven’t looked back.

I was able to export the Wordpress site with a Wordpress to Jekyll Exporter. The Octopress site has suggestions for hosting, one of which was Heroku. Since I’m new to markdown formatting, I looked for a suitable editor for my Mac. LightPaper seems to work really well and has a nice side-by-side view with a preview of the formatting.

UPDATE: Shortly after publishing this blog post, I found directions for hosting static HTML on Amazon S3. You have to pay for the storage, and the load balancing, but in return you get a globally load-balanced static site for a couple of bucks a month. Heroku has been working great (for free) so I won’t be doing this anytime soon. It’s nice to know it’s an option though.

File Stash

Hosting my own storage when free services are ubiquitous is silly. I set up a free account on Dropbox and as a bonus keep everything in sync between phone, iPad, and Mac.

Proxy Server

I don’t have a replacement for this yet, and it’s not a huge priority. For untrusted wifi I can fire up my AWS instance temporarily. My new employer has much tougher firewall/proxy/etc. rules and I don’t particularly want to get in trouble with HR attempting to circumvent them — tethering to the cell network works just fine if I need it.

I see a few options to explore:

  • Host something at home and use a dynamic dns service
  • A small virtual server from prgmr (or similar). I used to have a prgmr VPS and would happily go back
  • A commercial VPN service

Ultimately I would probably like to go with the commercial VPN service — less for me to maintain and configure, and (if I choose the right service) I can choose which country the VPN terminates in — useful for circumventing geographic restrictions…

UPDATE: Hosting something at home turned out to be incredibly simple. no-ip works with my router and I had an old laptop laying around to use. I still haven’t gotten the VPN functional yet though — wifi at work has been down due to a PoE issue so it has been a moot point.

Limitations

A downside to using all of these cloud-hosted / free services imposes a lack of privacy. For my public blog this isn’t an issue. For my personal communications and files it is a problem.

For the free services I am using, such as Heroku there is the risk that the free tier will be eliminated or that the performance will be inadequate.

For the paid services I use or anticipate using, the costs can add up rapidly. The convenience of not having to self-host an alternative needs to be weighed against the cost of the service.

Setting Up an Initial Erlang/Emacs Environment on Windows

| Comments

I’m just starting to learn Erlang and have figured out that Emacs is a pretty popular choice for an editor/IDE. For good reason — the syntax highlighting is excellent, you can have the erlang shell open within Emacs, so you don’t have to edit->save->switch to Erlang->compile->try your code. Instead you hit ctrl-c, ctrl-k and you’re dumped in an erlang shell with the code already compiled. If there were errors you can jump back to the offending line with another keystroke.

Unfortunately, getting everything set up is not simply clicking “OK” on an automated installer dialogue. But it’s not terribly hard. The hard part is getting used to Emacs if you’re new to it — and if you are new to Emacs, be assured that it is worth spending the time to learn it.

Here is what I’ve found:

Setting up an Erlang environment on Windows 7:

1: Download Erlang http://www.erlang.org/download.html Pick the 64 bit binary file (or the appropriate one for your version of windows). Run the installer.

2: Download Emacs  http://www.gnu.org/software/emacs/, http://ftpmirror.gnu.org/emacs/.

Pick the latest .zip file for windows.

Expand the zip file into c:\emacs

Browse into c:\emacs\emacs\bin

Run “addpm.exe”

Open your start menu, browse to the emacs icon, right-click on it. Set the start directory to someplace you’d like it to be (e.g. when opening files, the directory you start in — c:\dev, or c:\users\username\Documents\, or whatever).

Go into computer properties and set a new user variable “HOME” = c:\users\username\

Edit c:\users\username.emacs, and put the following into it: [Note the “.” before the emacs filename]

(setq load-path (cons  “C:/Program Files/erl5.10.3/lib/tools-2.6.12/emacs”       load-path))

(setq erlang-root-dir “C:/Program Files/erl5.10.3″)

(setq exec-path (cons “C:/Program Files/erl5.10.3/bin” exec-path))

(require ‘erlang-start)

[Note: Modify the version numbers to match what you’ve got.]

3: Run erlang from the start menu icon to ensure it comes up.

4: Run emacs from the start menu, then open/create an erlang (.erl) file. Type in some erlang and make sure the syntax highlighting is on.

Erlang mode in Emacs is described here: http://www.erlang.org/doc/man/erlang.el.html and http://www.erlang.org/doc/apps/tools/erlang_mode_chapter.html

Firefox Proxy Jumping Through Two Ssh Hosts

| Comments

Situation: You want unrestricted web access but are behind a firewall that doesn’t allow direct web access.

You can follow well-known recipes to use ssh or putty as a socks-5 proxy for firefox by ssh’ing to an outside server. But what if your firewall also blocks tcp/22 (ssh) to the Internet? If you have access to a *nix box that you can ssh to and that can in turn ssh out to the Internet, you can proxy through that.

Multihop SSH with Putty has the necessary steps to get this working. Basically you:

  • Set up ssh keys to access both the outside and inside hosts with key-based access.
  • Create a putty profile that specifies the inside server and it’s associated keyfile (with no tunnels defined), and save it.
  • Create another putty profile that specifies the outside server and the keyfile for the outside server, with some extra configuration:
  • Define the tunnel (for example D 9870)

  • On the proxy portion set the proxy command line to: plink.exe -load putty-profile-name-saved-for-first -nc  %host:%port\n

  • Set the “Proxy type” to “local”
  • Clear the proxy hostname
  • Save the second putty profile.
  • Test. Load up firefox, set the socks-5 proxy to localhost:9870 (make sure to clear all other proxy fields). Load up putty and load the second profile you created. It should dump you at a command shell on the outside server.
  • In firefox go to whatismyip.com — you should see the IP of your outside machine.

If it is all working correctly, your local network people only see an ssh connection from your workstation to the internal server and from the internal server to your external box.

Warning: DNS requests will still be “leaking” locally. To eliminate those you need to set up privoxy (or similar) on your outside box, then tunnel to that instead of using socks, with an http, https, and ftp proxy setting.

 

 

Creating a Library of Re-usable Code Snippets

| Comments

I often have to tackle a task that requires:

  • Extracting/parsing data from some tool or log file
  • Running a command on a list of systems
  • Running a command I rarely use with complicated syntax

Figuring these out is simple enough, but inevitably I get an email months or years later wanting me to repeat the same thing. Hmmm… How did I do that before?

My solution to this is fairly simple. Genuine throwaways where just knowing the syntax or that the command exists I dump in a “code snippets” page in OneNote. This works with any note-taking app or even a plain text file.

For example:
Need joeware adfind app:
adfind -nodn -default -csv -f “(&(objectClass=user)(objectCategory=person)(!userAccountControl:AND:=2))” samAccountName  msExchHomeServerName >
mailboxes.csv
– Finds accounts that are not disabled, and lists where their exchange mailbox is.

Maybe you have the userAccountControl bit flags memorized and can rattle this off with no trouble. For the rest of us, having this snippet is handy.

For more complicated tasks I create a little “project” directory. Within that I put a series of scripts, named in the sequence they need to run. If there is a data file source I generally copy that in as well. When I’ve got everything working as desired I’ll throw in a README.txt documenting what it does, how to generate the source (if that isn’t the “a” script), or where to get it, and list any useful variations. So when that email rolls around six months from now I can re-run the job the same way. I often re-use parts of the same scripts over and over — for example the “for /f” syntax for a Windows CMD prompt.

For example: “Fix computer description with name” was created during a Windows 7 rollout, where we wanted the computer description in AD to list the users name (computer names were just a number like 20993).

The scripts/files are:

  • a.pl
  • b.cmd
  • readme.txt
  • go.cmd

a.pl is fed a CSV file with a user name and the associated computer name, and a second csv file that is created from a dsquery command. It then outputs “b.cmd” a command script that has a separate “admod” command for each object I want to update.
The readme.txt documents the full process.
“go.cmd” triggers the process.

This is a really quick-and-dirty way of getting this task done, and there are many alternatives. I could have (for example):

  • Written this as a self-contained powershell, vbscript, perl, python, ruby, etc. script. However I already know the dsquery and admod commands I want to run, so why re-invent those wheels?
  • Found some way to do it in a GUI tool like Hyena. However the effort needed to do this is probably similar to just doing the quick and dirty method.
  • Strung everything together (ala go.cmd above) so that one command starts chewing away at changing AD entries. This is dangerous, as any mistake could lead to very unintended consequences. I maintain some degree of safety by outputting a “b.cmd” file that I can then review before running. This can then be saved for future reference, so I know when specific entries were changed.

This last point is really the crux of this technique: incremental scripts, none of which do a lot on their own, but that can be debugged seperately without causing major problems.

I have quite a library of these “scriptlets” now. They all solved some specific issues and may someday help solve new ones. The a, b, c, etc. scheme has gotten very long at times, and the languages are sometimes perl (my favorite), cmd, bash, powershell, python, or some hideous mix of them all.

I am also not afraid to output a CSV file where final cleanup is done in a spreadsheet such as Excel. Perl lets me do 99% of the heavy lifting, and Excel lets me format it nicely, add charts, etc.

Figuring Out AWS’s Free Tier and Pricing

| Comments

I signed up for a free Amazon Web Service account a few months ago and spun up a generic Linux box. With Google Reader going away I figured that would be a great place to host my tt-rss system, so configured that. A week or two later I noticed a whopping $0.16 balance on my AWS account, all due to excessive disk I/O.

Further research showed I could have a free RDS (database) instance with a lot more I/Os included in the free tier for usage. Switching over from MySQL on “my” box was surprisingly easy, the biggest snag was realizing I had to have both the MySQL RDS and Linux EC2 hosts in the same security group. Perhaps unsurprisingly, hosting the MySQL off of my micro instance boosted performance considerably.

A few days have gone by and I looked at my account activity and usage. I’ve got a $0.18 balance now, but almost all of my I/O has shifted off of the EC2 instance. The dropoff is pretty dramatic as seen in this chart:

EC2_write_IOPS