The curious case of AWS_DEFAULT_PROFILE

If you have multiple AWS accounts and use Amazon’s command line interface, you are probably already familiar with setting up multiple profiles. The benefit to this is that multiple profiles are respected in tools that use Amazon’s SDK, for the most part.

Which brings us to AWS_DEFAULT_PROFILE. If you read CLI documentation, you’ll see that the way you are apparently supposed to switch profiles is by using the AWS_DEFAULT_PROFILE variable, e.g. AWS_DEFAULT_PROFILE=staging.

This is wrong.

If you use boto (or other tools that employ it, such as ansible), you may be surprised to find that setting this variable does nothing, depending on what versions of the tool you are using. Tools such as boto and node are standardizing on using AWS_PROFILE instead, e.g. AWS_PROFILE=staging. (For example, see this thread on boto or Amazon’s document on configuring node.) The AWS CLI supports this too, so there’s no reason to continue to use AWS_DEFAULT_PROFILE.

Tagged with: , , ,
Posted in Troubleshooting

Fix SSL Error: Cannot connect to the real

A co-worker started getting the following error on her Mac running Mac OS X 10.7.5:


For some reason, the SSL certificate chain was no longer valid. After checking to make sure there were no Apple software updates, Chrome was up to date, no proxying, and the date / time was correct, I took a look at the certificate chain. It claimed that the root certificate had expired, but there wasn’t an update for it.

Based on the helpful comments on this article, I found this page on DigiCert root certificates.

I had her visit the Test page for the DigiCert High Assurance EV Root CA certificate, and as expected, it did not work properly for her. I had her download the appropriate root certificate, and she was back in action.

I figure this might be due to a lack of vendor updates around this version of Mac (10.7.x may be out of cycle now), so think she probably will need an update to Mavericks. In the mean time, she’s unblocked!

Tagged with: ,
Posted in How To

Hacking up varnish logging on Ubuntu 12.04

Varnish is an excellent page cache / web accelerator program that is well worth a look. Quick, where does varnish log by default? Trick question – by default Varnish doesn’t log anywhere. Let’s explore this a little bit.

Using varnishncsa for live logging

The first place you might look is just looking at the live data. Varnish provides the utility varnishncsa to view logs live in NCSA common log format. To fire it up in a console, just run varnishncsa. You can change the format, log it to a file, daemonize it…check the documentation for more.

Running varnishncsa on boot

Running Ubuntu 12.04 or newer? Piece of cake. Just edit /etc/default/varnishncsa, and set VARNISHNCSA_ENABLED=1. Then run: sudo /etc/init.d/varnishncsa start

Going Deeper: Logging X-Forwarded-For and Change Log Directory

Now is when we get to the hacky part. What if we want to change the logging format or change the log directory? Alas – the only way to do this is to edit the init script. Here are a few things I want to edit:

  • Log X-Forwarded-For
  • Custom Logging
  • Change log directory

Log X-Forwarded-For

Running behind a reverse proxy? Using an Elastic Load Balancer? The IP address in the log will not be of the end user, but of the reverse proxy or ELB. The end user IP will be in an HTTP header called “X-Forwarded-For”. If you’ve looked at the varnishncsa documentation, you know you can set this in two ways: use -f to “prefer X-Forwarded-For HTTP header over client.ip in the log output”, or use -F and specify a custom log format. If doing the former, this is really easy. Edit /etc/init.d/varnishncsa, and look for this line:


Change this by appending -f:


Custom Logging

Don’t like the NCSA custom log format? Want to add a field? Create your own format. However, this is a little non-intuitive. Create the format at the top of the init script, say, after defining the USER. You can get a list of the formatting options from the varnishncsa documentation.

For instance, let’s say we want to add the time before the first byte is received from the backend. This will help track how much time it takes to get a response from the backend, which could explain latencies. I’ll also want to keep the X-Forwarded-For IP instead of client IP.

I’ll just append it to the default format. So, my format string would be:

%{X-Forwarded-For}i %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" %{Varnish:time_firstbyte}x

Remove the -f option from DAEMON_OPTS if you added it above; we’re going to be writing our own format. At the top in the variable definition section, let’s add a log format:

DESC="HTTP accelerator log deamon"
LOGFORMAT="%{X-Forwarded-For}i %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" %{Varnish:time_firstbyte}x"

How do we get the format into the daemon start? Edit the DAEMON_OPTS line to add -F, but do NOT add the LOGFORMAT. It won’t work, trust me.


Scroll down to the start_varnishncsa() code block, and after ${DAEMON_OPTS}, add ${LOGFORMAT}. E.g.:

start_varnishncsa() {
output=$(/bin/tempfile -s.varnish)
log_daemon_msg "Starting $DESC" "$NAME"
if start-stop-daemon --start --quiet --pidfile ${PIDFILE} \
--chuid $USER --exec ${DAEMON} -- ${DAEMON_OPTS} \
> ${output} 2>&1; then
log_end_msg 0
log_end_msg 1
cat $output
exit 1
rm $output

Change log directory

The first part of this is relatively straightforward. Let’s say we have a separate partition for logging, at say, /mnt/logs. We want to log to /mnt/logs/varnish instead of the default /var/log/varnish. Edit /etc/init.d/varnishncsa and set the LOGFILE variable:

DESC="HTTP accelerator log deamon"

That’s all fine and good, but what if you don’t have permissions to that directory? Let’s fix that too. Above the start_varnishncsa() block, let’s add the following:

# Fix permissions for custom logging
if ! [ "$(stat -c '%U:%G' $(dirname ${LOGFILE}))" = "varnishlog:varnishlog" ]; then
chown -R varnishlog:varnishlog $(dirname ${LOGFILE});

This way, if the log directory and its contents are not owned by varnishlog, it’ll fix this. Note that placement here is after checking VARNISHNCSA_ENABLED and that the DAEMON exists, so it’ll only run if we’ve passed those checks.

Almost there! One other snag is that log rotation will not rotate this log. Edit /etc/logrotate.d/varnish and add the path:

/var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log \
/mnt/logs/varnish/varnishncsa.log {
rotate 31
for service in varnishlog varnishncsa; do
if /usr/bin/pgrep -P 1 $service >/dev/null; then
/usr/sbin/invoke-rc.d $service reload > /dev/null


Varnish logging isn’t the most straightforward and obvious. The above works, but is kind of kludgy: we’ve dumped a lot of stuff into the init script, and it’s highly custom to this particular installation. I created a modified version of the above that stuffs the configuration information into /etc/default/varnishncsa, but that also doesn’t feel quite right. I’d ideally like to edit, say, /etc/varnish/varnishncsa.cfg or something, but this will do. Though it’s somewhat non-intuitive, varnish is worth the effort.

Tagged with: , ,
Posted in How To

How to kill a lot of screen sessions at once

I use screen pretty much every day, but sometimes I end up orphaning a lot of screen sessions. (For instance, why is it Mac disables your Internet if you lock your screen? STOP THAT.)

Anyway, at one point I had 18 screen sessions running detached. How do you destroy them? You could re-attach each session, then issue a Cntrl-x \ , but that would quickly grow repetitive.

The way to kill a screen session is to run: screen -X -S PID quit

Copy/pasting all that sounds tedious too. Fine, let’s do it the sysadmin-y way:

  • Get the list of screens
  • grep for Detached
  • cut for the secondĀ field
  • apply command.

Piece of cake!

for x in $(screen -ls | grep Detached | cut -f2); do screen -X -S $x quit; done

Ta da! Power screen kill! I find this example interesting in the sense of the right way to approach problems: don’t necessarily just do work by hand; give a little thought to how to do things better.

The irony that the effort I saved is lost on creating this blog post is not lost on me. However, it emphasizes a good secondary skill: document what you do. Though it may take even more time, it’ll pay off in the future. It might not for the particular thing you’re documenting, but as a habit, it’s invaluable.

Tagged with:
Posted in How To

How to Drop a MongoDB Collection With An Illegal Name

For whatever reason, you may have gotten collections in MongoDB that you cannot access or delete, due to their name. Maybe you led the collection name with a number, instead of a letter. Your collection name may be illegal or inaccessible for any of a variety of reasons. In my case, I had collections with names like the following:

In this case, it’s likely the dashes are causing issues. That’s a mouthful, so I’ll just refer to that string as ILLEGALLYNAMEDCOLLECTION. However, what’s more interesting is how you access that data. You can’t count it like so:

Wed May 22 12:58:45.932 JavaScript execution failed: SyntaxError: Unexpected token ILLEGAL

You also can’t drop() it that way. How do you access it? Use getCollection.

> db.getCollection("ILLEGALLYNAMEDCOLLECTION").count()
> db.getCollection("ILLEGALLYNAMEDCOLLECTION").drop()

So long, illegally named collection!

Tagged with:
Posted in How To

Rehash For Bash

Changed the path to an executable? If the old path is in your path, you need to rebuild the hash of paths. On tcsh, you did this by running ‘rehash’. On bash, run:

hash -r

What if you want to turn off hashing completely? Run the following, or add to your .bashrc if you want it permanent:

set +h
Tagged with: ,
Posted in How To

Repack a Debian Package

Got a debian package that is so old, it’s no longer found in repositories? Need a specific version from a server, but don’t have the original deb file? Repack it with dpkg-repack.

sudo apt-get install dpkg-repack

Voila – instantly recreated package. Now fix your code so you can run current software again.

Tagged with: ,
Posted in How To