Balaji Vajjala's Blog

A DevOps Blog from Trenches

Code Reuse With Node.js

Code Reuse With Node.js

Code recycling

Any project that grows to a decent size will need to re-use parts of its code extensively. That often means, through the development cycle, a fair amount of rewrites and refactoring exercises. Elegant code re-use is hard to pull off.

With node.js, which we use quite a bit at devo.ps, the most common ways to do this often rely on prototype or class inheritance. The problem is, as the inheritance chain grows, managing attributes and functions can become quite complex.

The truth is, people usually just need the objects. This led us to adopt a certain form of object-based prototyping. We believe it to be leaner and more straightforward in most cases. But before we get there, let’s have a look at how people usually approach this issue.

The “Function copy”

Usually in the form of this[key] = that[key]. A quick example:

var objectA = {
    lorem: 'lorem ipsum'
};
var objectB = {};

// Direct copy of a string, but you get the idea
objectB.lorem = objectA.lorem;
console.log(objectB); // Will output: { lorem: 'lorem ipsum' }

Crude, but it works. Next…

Object.defineProperties()

The previous method may work with simple structures, but it won’t hold when your use cases become more complex. That’s when I usually call my buddy Object.defineProperties():

var descriptor = Object.getOwnPropertyDescriptor;
var defineProp = Object.defineProperty;

var objectA = {};
var objectB = {};
var objectC = {};

objectA.__defineGetter__('lorem', function() {
    return 'lorem ipsum';
});
console.log(objectA); // Will output: { lorem: [Getter] }

// Direct copy, which copies the result of the getter.
objectB.lorem = objectA.lorem;
console.log(objectB); // Will output: { lorem: 'lorem ipsum' }

// Copying with Object.defineProperty(), and it copies the getter itself.
defineProp(objectC, 'lorem', descriptor(objectA, 'lorem'));
console.log(objectC); // Will output: { lorem: [Getter] }

I often use a library for that. A couple examples (more or less the same stuff with different coding styles):

  1. es5-ext

    var extend = require(‘es5-ext/lib/Object/extend-properties’);

    var objectA = {}; var objectC = {};

    objectA.defineGetter(‘lorem’, function() { return ‘lorem ipsum’; });

    extend(objectC, objectA); console.log(objectC); // Will output: { lorem: [Getter] }

  2. Carcass

    var carcass = require(‘carcass’);

    var objectA = {}; var objectC = {};

    objectA.defineGetter(‘lorem’, function() { return ‘lorem ipsum’; });

    carcass.mixable(objectC); objectC.mixin(objectA); console.log(objectC); // Will output: { mixin: [Function: mixin], lorem: [Getter] }

Slightly better, but not optimal. Now, let’s see what we end up doing more and more often:

Prototyping through objects

The basic idea is that we prepare some functions, wrap them into an object which then becomes a “feature”. That feature can then be re-used by simply merging it with the targeted structure (object or prototype).

Let’s take the example of the loaderSync script in Carcass:

module.exports = {
    source: source,
    parser: parser,
    reload: reload,
    get: get
};

function get() {

(...)

Once you copy the functions to an object, this object becomes a “loader” that can load a “source” synchronously with a “parser”. A “source” can be a file path and the “parser” can be simply Node.js’ require function.

Let’s now see how to use this with a couple object builders. Once again, I’ll borrow an example from Carcass; the loaderSync benchmark script. The first builder generates a function and copies the methods from what we’ve prepared. The second one copies the methods to the prototype of a builder class:

(...)

function LoaderA(_source) {
    function loader() {
        return loader.get();
    }
    loader.mixin = mixin;
    loader.mixin(loaderSync);
    loader.source(_source);
    return loader;
}

(...)

function LoaderC(_source) {
    if (!(this instanceof LoaderC)) return new LoaderC(_source);
    this.source(_source);
}
LoaderC.prototype.mixin = mixin;
LoaderC.prototype.mixin(loaderSync);

(...)

Here we can see the two approaches. Let’s compare them quickly:

FeatureLoader ALoader C

Instantiating var a = LoaderA(...)

var c = LoaderC(...) or var c = new LoaderC(...)

Appearance Generates a function

Builds a typical instance which is an object.

Invoking directly a() or a.get()

c.get()

Invoking as a callback ipsum(a)

ipsum(c.get.bind(c))

Performance † of instantiating

100x faster

Performance of invoking idem

idem

: (check it yourself by benchmarking Carcass with make bm)

“Protos” and beyond

That last approach is gaining traction among our team; we prepare functions for our object builders (which, by the way, we call “protos”). While we still choose to use prototypes in some occurrences, it is mainly because it is faster to get done. For the sake of convenience, we also sometimes rely on functions rather than objects to invoke our “protos”, however keep in mind that this is a performance trade-off.

I’ll wrap this up mentioning one more method we use, admittedly less often: “Object alter”. The idea is to rely on an “alter” function designed to change objects passed to it. This is sometimes also called a “mixin”. An example from vsionmedia’s trove of awesomeness on Github:

(...)

module.exports = function(obj){

    obj.settings = {};

    obj.set = function(name, val){
        (...)
    };

    (...)

    return obj;
};

Resources

Automation And Friction

Automation And Friction

I’ll admit that the devo.ps team is a lazy bunch; we like to forget about things, especially the hard stuff. Dealing with a complex process invariably leads one of us to vent about how “we should automate that stuff”. That’s what our team does day and night:

  1. Dumb things down, lower barriers of entry, and then…
  2. Automate all the things!

This has transpired through every layer of our company, from engineering to operations. Recently we’ve started pushing on a third point, but first let me rant a bit…

The ever increasing surface of friction

The past few years have seen a healthy push on UI and UX. Even developer tools and enterprise software, historically less user-friendly, have started adopting that trend. We now have things like Github. Great.

This trend grew in parallel with the adoption of SaaS. SaaS are the results of teams focused on specific problems, with the user experience often being a key component (not to undervalue good engineering). It’s pretty standard for these services to offer an API for integration’s sake. Our CRM plays nicely with Dropbox, GMail and a gazillion other services. Again, great.

However, the success of SaaS means the surface of interfaces we’re dealing with is constantly stretching. This is far more difficult to overcome than poor UI or UX. Many of us have witnessed teams struggling to get adoption on a great tool that happen to be one too many. There’s not much you can do about it.

A bot to rule them all…

Borat is omnipotent

Our team has tried a lot of different approaches over the years. We kicked the tires on a lot of products and ended up doing as usual:

  1. Simplify. For example, we use Github to manage most tasks and discussions, including operations (HR, admin, …), and marketing. We used Trello alongside Github for a while and we loved it. But it silo-ed the discussions. Everything from our employee handbook to tasks for buying snacks for the office are now on Github. It also had an interesting side effect on transparency, but I’ll talk about this another time.

  2. Automate. We automate pretty much everything we can. When you apply to one of our job by email for example, we push the attachments in Dropbox (likely your resume) and create a ticket with the relevant information on Github. Zapier is great for this kind of stuff by the way.

  3. Make it accessible. That’s the most important point for us at this stage. Borat, our Hubot chat bot, is hooked up with most of our infrastructure and is able to pass on requests to the services we use as well as some of our automation. If one of us is awake, chances are you can find us on the chat, making it the most ubiquitous interface for our team:

    • Need to deploy some code on production or modify some configuration on a server? Ask Borat, he’ll relay your demands to the devo.ps API.
    • Your latest commit broke the build? A new mail came from support? Expect to hear about it from Borat.
    • Need to use our time tracker? Just drop a message to the bot when you’re starting your task and let him know when you’re done.
    • Need to call for a SCRUM? Just mention the Github team you want to chat with and Borat will create a separate channel and invite the right people to join.
    • Somebody is at the door? Ask the bot to open it for you (you gotta love hacking on Raspberry PI).

Anybody with access to our bot’s repository can add a script to hook him up to a new service. Git push, kill the bot and wait for him to come back to life with new skills. The tedious stuff ends up sooner or later scripted and one sentence away.

Really, try it. It’s worth the investment.

Farewell to Regular Web Development Approaches

Farewell to Regular Web Development Approaches

At my previous company, we built Web applications for medium to large organizations, often in the humanitarian and non-profit space, facing original problems revolving around data. Things like building the voting infrastructure for the Southern Sudan Referendum helped us diversify our technical chops. But until a year ago, we were still mostly building regular Web applications; the user requests a page that we build and serve back.

Until we started falling in love with APIs and static clients.

It’s not that we fundamentally disliked the previous approach. We just reached a point where we felt our goals weren’t best served by this model. With lots of dynamic data, complex visualizations and a set of “static” interfaces, the traditional way was hindering our development speed and our ability to experiment. And so we got busy, experimenting at first with smaller parts of our projects (blog, help section, download pages…). We realized our use of complex pieces of softwares like content management systems had seriously biased our approach to problem solving. The CMS had become a boilerplate, an unchallenged dependency.

We’ve gradually adopted a pattern of building front-ends as static clients (may they be Web, mobile or 3rd party integrations) combined with, usually, one RESTful JSON API in the backend. And it works marvelously, thanks in part to some awesome tech much smarter people figured out for us:

Most of what we build at devo.ps is stemming from this accelerated recovery and follow usually that order:

  1. We start by defining our API interface through user stories and methods,
  2. Both backend and front-end teams are getting cranking on implementing and solving the challenges specific to their part,

A lot of things happen in parallel and changes on one side rarely impact the other: we can make drastic changes in the UI without any change on the backend. And there were a lot of unexpected gains too, in security, speed and overall maintainability. More importantly, we’ve freed a lot of our resources to focus on building compelling user experiences instead of fighting a large piece of software to do what we want it to do.

If you haven’t tried building things that way yet, give it a try honestly (when relevant); it may be a larger initial investment in some cases but you’ll come out on top at almost every level.

Why We Dropped Swagger And I/O Docs

Why We Dropped Swagger And I/O Docs

As we started investing in our new strategy at my previous company, we looked around for solutions to document APIs. It may not be the sexiest part of the project, but documentation is the first step to designing a good API. And I mean first as in “before you even start writing tests” (yes, you should be writing tests first too).

We originally went with a simple Wiki page on Github, which served us just fine in the past. But it quickly became clear that it wasn’t going to cut it. We started thinking about what good documentations is. We’re fans of the single page approach that the Backbone.js documentation illustrates well and clearly remembered Github and Stripe as easy and well organized resources. Some Googling later, we were contemplating Wordnik’s Swagger and Mashery’s I/O Docs. We later settled for I/O Docs as it is built with node.js and was more straightforward to set up (for us at least).

Once again, we hit a wall with this approach:

  1. No proper support for JSON body: we don’t do much with parameters and mostly send JSON objects in the body of our requests, using HTTP verbs for the different types of operations we perform on our collections and models in the backend. Swagger and I/O Docs fall short of support for it, letting you simply dump your JSON in a field: not ideal.

  2. You’re querying the actual API: to be fair, this is an intentional feature. Now some of you may find it interesting that your documentation allows users to easily run calls against your API. That’s what Flickr does with their API explorer, and we used to think it was pretty neat. But once we started using it, we saw the risks of exposing so casually API calls that can impact your platform (especially with devo.ps which deals with your actual infrastructure). I guess you could set up a testing API for that very purpose, but that’s quite a bit of added complexity (and we’re lazy).

And that’s how we ended up putting together Carte, a very lightweight Jekyll-based solution: drop a new post for each API call, following some loose format and specifying a few bits of meta data in the YAML header (type of the method, path…) and you’re good to go.

Screenshot of Carte

We’re real suckers for Jekyll. We’ve actually used it to build quite a few static clients for our APIs. One of the advantages of this approach is that we can bundle our documentation with our codebase by simply pushing it on the gh-pages branch, and it pops up as a Github page. That’s tremendously important for us as it make it very easy for developers to keep the documentation and the code in synch.

Carte is intentionally crude: have a look at the README and hack at will. Drop us a shout at @devo_ps if you need help or want to suggest a feature.

Best Practices: It’s Always or Never

Best Practices: It’s Always Or Never ( And Preferably Always)

Messy cables

It’s Monday morning. The development team needs a box and you’re already contemplating the gazillion other urgent tasks that need to be done on the existing infrastructure. _Just that one time_TM, you’re going to forget about your own rules. You’re just gonna spawn an instance, set up the few services needed and be done with it. You’ll drop some of the usual time suckers: backup strategy, access rules, init scripts, documentation… You can’t just do the whole of it AND handle the rest of your day-to-day responsibilities. After all, it’s just a development server and you’ll probably fold it in a couple weeks, or you’ll clean it up once your plate is a tad less full.

A few weeks later, the box is still there and your backlog is far from looking less crowded. The development team just rolled out their production application on the same box. And things start crashing… badly.

After a couple of not so courteous emails from the dev team mentioning repetitive crashes, you log in the box and the fun starts. You can’t figure out what services have been deployed, or how exactly they were installed. You can’t restore the database because you don’t know where the bloody backups are. You waste time to find out that CouchDB wasn’t started at boot. All of this while receiving emails of “encouragement” from your colleagues.

Just because of that “one time”. Except that it’s never just that one time.

Best practices are not freaking optional

I hear you: coming up with these best practices and sticking to it systematically is hard. It’s high investment. But based on our common experience, it’s one you can’t afford not making. The “quick and dirty that one time” approach will ultimately fail you.

A few things you should never consider skipping:

  • Document the hell out of everything as you go. You probably won’t have time to get it done once you shipped it, and you probably won’t remember what you did or why you did it in a few weeks from now. Your colleagues will probably appreciate too.

  • Off-site backups for everything. Don’t even think of keeping your backups on the same physical box. Disks fail (a lot) and storage like S3/Glacier is dirt cheap. Find out a way to backup your code and data and stick to it.

  • Full setup and reliable sources. Avoid random AWS AMIs or RPM repositories. And when settings things up, go through the whole shebang: init script, dedicated running user, environment variables and such are not optional. Some of us also think that you shouldn’t use rc.local for your Web services ever again.

Infrastructure As Code And Automation

Obviously, given what we’re working on at devo.ps, we’re pretty strong adopters of infrastructure as code and automation. What tools to use is a much larger discussion. Go have a look at the comments on the announcement of the new version of Chef to get an idea of what’s out there.

Ultimately these are just opinions, but behind them are concepts worth investing in. Capturing the work you do on your infrastructure in repeatable and testable code, and automating as much as you can helps removing yourself from the equation. Doing so is helping you to reduce the human factor and free yourself of the repetitive boilerplate while you focus on the challenging tasks that only a creative brain can tackle.

Not building upon best practices is simply not an option. By doing so, you fail at investing in the foundation for a more robust infrastructure, and more importantly it is depriving you from scaling yourself.

Picture from comedy_nose

Troubleshooting 5minutes on a yet Unknown Box

First 5 Minutes Troubleshooting A Server

Back when our team was dealing with operations, optimization and scalability at our previous company, we had our fair share of troubleshooting poorly performing applications and infrastructures of various sizes, often large (think CNN or the World Bank). Tight deadlines, “exotic” technical stacks and lack of information usually made for memorable experiences.

The cause of the issues was rarely obvious: here are a few things we usually got started with.

Get some context

Don’t rush on the servers just yet, you need to figure out how much is already known about the server and the specifics of the issues. You don’t want to waste your time (trouble) shooting in the dark.

A few “must have”:

  • What exactly are the symptoms of the issue? Unresponsiveness? Errors?
  • When did the problem start being noticed?
  • Is it reproducible?
  • Any pattern (e.g. happens every hour)?
  • What were the latest changes on the platform (code, servers, stack)?
  • Does it affect a specific user segment (logged in, logged out, geographically located…)?
  • Is there any documentation for the architecture (physical and logical)?
  • Is there a monitoring platform? Munin, Zabbix, Nagios, New Relic… Anything will do.
  • Any (centralized) logs?. Loggly, Airbrake, Graylog…

The last two ones are the most convenient sources of information, but don’t expect too much: they’re also the ones usually painfully absent. Tough luck, make a note to get this corrected and move on.

Who’s there?

$ w
$ last

Not critical, but you’d rather not be troubleshooting a platform others are playing with. One cook in the kitchen is enough.

What was previously done?

$ history

Always a good thing to look at; combined with the knowledge of who was on the box earlier on. Be responsible by all means, being admin shouldn’t allow you to break ones privacy.

A quick mental note for later, you may want to update the environment variable HISTTIMEFORMAT to keep track of the time those commands were ran. Nothing is more frustrating than investigating an outdated list of commands…

What is running?

$ pstree -a
$ ps aux

While ps aux tends to be pretty verbose, pstree -a gives you a nice condensed view of what is running and who called what.

Listening services

$ netstat -ntlp
$ netstat -nulp
$ netstat -nxlp

I tend to prefer running them separately, mainly because I don’t like looking at all the services at the same time. netstat -nalp will do to though. Even then, I’d ommit the numeric option (IPs are more readable IMHO).

Identify the running services and whether they’re expected to be running or not. Look for the various listening ports. You can always match the PID of the process with the output of ps aux; this can be quite useful especially when you end up with 2 or 3 Java or Erlang processes running concurrently.

We usual prefer to have more or less specialized boxes, with a low number of services running on each one of them. If you see 3 dozens of listening ports you probably should make a mental note of investigating this further and see what can be cleaned up or reorganized.

CPU and RAM

$ free -m
$ uptime
$ top
$ htop

This should answer a few questions:

  • Any free RAM? Is it swapping?
  • Is there still some CPU left? How many CPU cores are available on the server? Is one of them overloaded?
  • What is causing the most load on the box? What is the load average?

Hardware

$ lspci
$ dmidecode
$ ethtool

There are still a lot of bare-metal servers out there, this should help with;

  • Identifying the RAID card (with BBU?), the CPU, the available memory slots. This may give you some hints on potential issues and/or performance improvements.
  • Is your NIC properly set? Are you running in half-duplex? In 10MBps? Any TX/RX errors?

IO Performances

$ iostat -kx 2
$ vmstat 2 10
$ mpstat 2 10
$ dstat --top-io --top-bio

Very useful commands to analyze the overall performances of your backend;

  • Checking the disk usage: has the box a filesystem/disk with 100% disk usage?
  • Is the swap currently in use (si/so)?
  • What is using the CPU: system? User? Stolen (VM)?
  • dstat is my all-time favorite. What is using the IO? Is MySQL sucking up the resources? Is it your PHP processes?

Mount points and filesystems

$ mount
$ cat /etc/fstab
$ vgs
$ pvs
$ lvs
$ df -h
$ lsof +D / /* beware not to kill your box */
  • How many filesystems are mounted?
  • Is there a dedicated filesystem for some of the services? (MySQL by any chance..?)
  • What are the filesystem mount options: noatime? default? Have some filesystem been re-mounted as read-only?
  • Do you have any disk space left?
  • Is there any big (deleted) files that haven’t been flushed yet?
  • Do you have room to extend a partition if disk space is an issue?

Kernel, interrupts and network usage

$ sysctl -a | grep ...
$ cat /proc/interrupts
$ cat /proc/net/ip_conntrack /* may take some time on busy servers */
$ netstat
$ ss -s
  • Are your IRQ properly balanced across the CPU? Or is one of the core overloaded because of network interrupts, raid card, …?
  • How much is swappinness set to? 60 is good enough for workstations, but when it come to servers this is generally a bad idea: you do not want your server to swap… ever. Otherwise your swapping process will be locked while data is read/written to the disk.
  • Is conntrack_max set to a high enough number to handle your traffic?
  • How long do you maintain TCP connections in the various states (TIME_WAIT, …)?
  • netstat can be a bit slow to display all the existing connections, you may want to use ss instead to get a summary.

Have a look at Linux TCP tuning for some more pointer as to how to tune your network stack.

System logs and kernel messages

$ dmesg
$ less /var/log/messages
$ less /var/log/secure
$ less /var/log/auth
  • Look for any error or warning messages; is it spitting issues about the number of connections in your conntrack being too high?
  • Do you see any hardware error, or filesystem error?
  • Can you correlate the time from those events with the information provided beforehand?

Cronjobs

$ ls /etc/cron* + cat
$ for user in $(cat /etc/passwd | cut -f1 -d:); do crontab -l -u $user; done
  • Is there any cron job that is running too often?
  • Is there some user’s cron that is “hidden” to the common eyes?
  • Was there a backup of some sort running at the time of the issue?

Application logs

There is a lot to analyze here, but it’s unlikely you’ll have time to be exhaustive at first. Focus on the obvious ones, for example in the case of a LAMP stack:

  • Apache & Nginx; chase down access and error logs, look for 5xx errors, look for possible limit_zone errors.
  • MySQL; look for errors in the mysql.log, trace of corrupted tables, innodb repair process in progress. Looks for slow logs and define if there is disk/index/query issues.
  • PHP-FPM; if you have php-slow logs on, dig in and try to find errors (php, mysql, memcache, …). If not, set it on.
  • Varnish; in varnishlog and varnishstat, check your hit/miss ratio. Are you missing some rules in your config that let end-users hit your backend instead?
  • HA-Proxy; what is your backend status? Are your health-checks successful? Do you hit your max queue size on the frontend or your backends?

Conclusion

After these first 5 minutes (give or take 10 minutes) you should have a better understanding of:

  • What is running.
  • Whether the issue seems to be related to IO/hardware/networking or configuration (bad code, kernel tuning, …).
  • Whether there’s a pattern you recognize: for example a bad use of the DB indexes, or too many apache workers.

You may even have found the actual root cause. If not, you should be in a good place to start digging further, with the knowledge that you’ve covered the obvious.

ZooKeeper vs. Doozer vs. Etcd

ZooKeeper vs. Doozer vs. Etcd

While devo.ps is fast approaching a public release, the team has been dealing with an increasingly complex infrastructure. We more recently faced an interesting issue; how do you share configuration across a cluster of servers? More importantly, how do you do so in a resilient, secure, easily deployable and speedy fashion?

That’s what got us to evaluate some of the options available out there; ZooKeeper, Doozer and etcd. These tools all solve similar sets of problems but their approach differ quite significantly. Since we spent some time evaluating them, we thought we’d share our findings.

ZooKeeper, the old dog

ZooKeeper is the most well known (and oldest) project we’ve looked into. It’s used by a few big players (Rackspace, Yahoo, eBay, Youtube) and is pretty mature.

It was created by Yahoo to deal with distributed systems applications. I strongly recommend you read the “making of” if you’re interested in understanding where Yahoo came from when they wrote it.

It stores variables in a structure similar to a file system, an approach that both Doozer and etcd still follow. With ZooKeeper, you maintain a cluster of servers communicating with each other that share the state of the distributed configuration data. Each cluster elects one “leader” and clients can connect to any of the servers within the cluster to retrieve the data. Zookeeper uses its own algorithm to handle distributed storage.

  • Pros:

    • Mature technology; it is used by some big players (eBay, Yahoo et al).
    • Feature-rich; lots of client bindings, tools, API…
  • Cons:

    • Complex; ZooKeeper is not for the faint of heart. It is pretty heavy and will require you to maintain a fairly large stack.
    • It’s… Java; not that we especially hate Java, but it is on the heavy side and introduce a lot of dependencies. We wanted to keep our machines as lean as possible and usually shy away from dependency heavy technologies.
    • Apache…; we have mixed feelings about the Apache Foundation. “Has Apache Lost Its Way?” summarizes it pretty well.

Doozer, kinda dead

Doozer was developed by Heroku a few years ago. It’s written in Go (yay!), which means it compiles into a single binary that runs without dependencies. On a side-note, if you’re writing code to manage infrastructure, you should spend some time learning Go.

Doozer got some initial excitement from the developer community but seems to have stalled more recently, with many forks being sporadically maintained and no active core development.

It is composed of a daemon and a client. Once you have at least one Doozer server up, you can add any number of servers and have clients get and set data by talking to any of the servers within that cluster.

It was one of the first practical implementations (as far as I know) of the Paxos algorithm). This means operations can be slow when compared to dealing with a straight database since cluster-wide consensus needs to be reached before committing any operation.

Doozer was a step in the right direction. It is simple to use and setup. However, after using it for a while we started noticing that a lot of its parts felt unfinished. Moreover, it wasn’t answering some of our needs very well (encryption and ACL).

  • Pros:

    • Easy to deploy, setup and use (Go, yay!)
    • It works; lots of people have actually used it in production.
  • Cons:

    • Pretty much dead: the core project hasn’t been active in a while (1 commit since May) and is pretty fragmented (150 forks…).
    • Security; no encryption and a fairly simple secure-word based authentication.
    • No ACL; and we badly needed this.

etcd

After experiencing the shortcomings of Doozer, we stumbled upon a new distributed configuration storage called etcd. It was first released by the CoreOS team a month ago.

Etcd and Doozer look pretty similar, at least on the surface. The most obvious technical difference is that ectd uses the Raft algorithm instead of Paxos. Raft is designed to be simpler and easier to implement than Paxos.

Etcd’s architecture is similar to Doozer’s. It does, however, store data persistently (writes log and snapshots), which was of value to us for some edge cases. It also has a better take on security, with CA’s, certs and private keys. While setting it up is not straightforward, it adds conveniency and safety of mind.

Beyond the fact that it answered some of our more advanced needs, we were seduced (and impressed) by the development pace of the project.

  • Pros:

    • Easy to deploy, setup and use (yay Go and yay HTTP interfaces!).
    • Data persistence.
    • Secure: encryption and authentication by private keys.
    • Good documentation (if a little bit obscure at times).
    • Planned ACL implementation.
  • Cons:

    • (Very) young project; interfaces are still moving pretty quickly.
    • Still not a perfect match, especially in the way that data is spread.

The DIY approach (yeah, right..?)

It is only fair that technical teams may rely on their understanding of their infrastructure and coding skills to get something that just works™ in place. We haven’t seriously considered this approach as we felt that getting security and distributed state sharing right was going to be a bigger endeavor than we could afford (the backlog is full enough for now).

Conclusion

In the end, we decided to give etcd a try. So far it seems to work well for our needs and the very active development pace seems to validate our choice. It has proven resilient and will likely hold well until we have the resources to either customize its data propagation approach, or build our own solution that will answer some needs it is not likely to answer (we’ve already looked into doing so with ZeroMQ and Go).

Deploy reveal.js slideshow on github-pages

Originally posted on Tikal Knolwedge’s webiste @ this link

Deploy reveal.js slideshow on github-pages was ridiculously easy.

Cloned reveal.js master branch to my local machine:

git clone git@github.com:hakimel/reveal.js.git

Remove the history – of you want to preserve history ( git remote rm origin and add your new remote)

rm -Rf .git 

Add your new remote (you need to create a repository first)

git remote add origin git@github.com:tikalk-cookbooks/chef_workshop_presentation.git

You know the rest …

git add .
git commit -m "Initial commit"
git push origin master

In order to push master to github pages you should create a branch names gh-pages and it must be named that, the commit & push to this branch and about 2 minuets later you can browse to your presentation on gh-pages.

Create a branch:

git branch hg-pages
git push origin gh-pages</pre>

updating this "site" can be done by editing the stuff you want on the "master" branch, then merge those changes in the gh-pages branch and finally push those changes to the remote gh-pages which will automatically deploy youe reveal.js presentaion.

the merge sould look somthing like:

hagzag-Lap-hagzag:chef_workshop_presentaion(master) $ git checkout gh-pages&nbsp;

hagzag-Lap-hagzag:chef_workshop_presentaion(gh-pages) $ git merge master&nbsp;
Updating 0f4f1e1..fb0d73d

Fast-forward

css/custom.css | &nbsp; 4 ++++

index.html &nbsp; &nbsp; | 104 +++++++++++++++++++++++++++++++++++++++++++++++++-----------    --------------------------------------------

2 files changed, 53 insertions(+), 55 deletions(-)

create mode 100644 css/custom.css

git push origin gh-pages

the url is built from the followng pattern:

[github_username].github.io/[repo_name]

in this case:

http://tikalk-cookbooks.github.io/chef_workshop_presentation/

Hope you find this useful !

HP

Manage Data Bags in Solo mode

For the past ~6 monthes or so I have been working solely with chef-solo. There are quite a few helpers for solo out there such as:

  1. soloist – https://github.com/mkocher/soloist
  2. knife-solo – https://github.com/matschaffer/knife-solo & a few more.

What kept annoying me for some time is I couldn’t manage databas with knife whilst working in solo mode … ARRRRRRGH!!!

It looked somthing like:

[oper@sandbox chef_repo]$ knife data bag create admins
ERROR: The object you are looking for could not be found
Response: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /data was not found on this server.</p>
<hr>
<address>Apache/2.2.15 (CentOS) Server at : Port 80</address>
</body></html>

Then came knife-solo_data_bag I am quite embaressed to say ;) I haven’t found this sooner. Now with a totally bogus knife.rb file I can generate / edit / manager databags with knife-solo & knife-solo_data_bag.

Again, hope you find this useful, I know I do ;)

HP

S3 - Mounting s3 buckets with S3fs

A cheap solution on Amazon s3 for uploading files to s3 via SSH

I needed to upload files from my Corporate environment in a push mode meaning I do not open any “special” ports in my environemnt in order to enable users to put files on S3. I am curentelly testing to see how robust this solution really is but basically what I found myself doing is install s3fs (link to project page) like so:

wget http://s3fs.googlecode.com/files/s3fs-1.73.tar.gz
tar xvzf s3fs-1.73.tar.gz
cd s3fs-1.73
./configure --prefix=/usr
make && make install

I was installing this for Suse but for Amazon linux/redhat etc you might find a package see: here.

Once the package is installed you can use s3fs

s3fs BUCKET:[PATH] MOUNTPOINT [OPTION]...

s3fs dev_tools /mnt/dev_tools/ 

Worth noting you do not need to specify the s3 url, you only specify the bucket name !

The same thing / very similir in /etc/fstab will look like so:

s3fs#dev_tools /mnt/dev_tools fuse allow_other,user=youruser 0 0

The mount opts are extreemly important – without the allow_other flag the user cannot write to the directory.

This is really awesome – we now jsut need to make sure the connectivity is reliable / fast enough and this will become very usefull.

As always hope you find this useful.