As part of a new job I’m taking, I wanted to learn more about image building for Openstack and other virtual environments. I’ve done it by hand for the customized OSes at my old job, but I haven’t had the chance to explore any automated solutions. I was pointed to Packer as a tool to build several different images at the same time (and automatically). It sounds like a great project and I’m going to use this post to get up to speed with using the basics. One quick caveat from the outset is that I’m not going to use Amazon at first. I’ll be running against my home Openstack lab since it’s free and a good excuse to get my homelab back in order.

##Install Packer##

I’ve got a shiny new Macbook, and installing Packer was actually really easy. The way I did it depended on homebrew, but you can also install manually from from their docs here.

  • In terminal, ensure that you have homebrew setup by issuing brew.
  • Add the necessary tap with brew tap homebrew/binary.
  • Finally, install packer with brew install packer.
  • You can test it’s installed by simply issuing packer in the terminal.

##Get Openstack Ready## I run an all-in-one deployment of RDO Openstack at home. Obviously, there’s a million different ways to deploy, but here and here are the pieces that I followed. It’s important to note that in my lab, instances come alive on a private network, then get access to my router’s 192.168.1.0/24 block via floating IPs. This will come in to play a bit later with the Packer template.

  • Get a known good image into Glance by importing one of the big distros. I used the Ubuntu 14.04 LTS image found here. You can just put that link into Glance’s import dialog. My final dialog looked like this:

  • Take note of the new image’s UUID, we’ll need that later:

[rsmitty@localhost ~(keystone_admin)]$ nova image-list
+--------------------------------------+-------------------+--------+----------+
| ID                                   | Name              | Status | Server
+--------------------------------------+-------------------+--------+----------+
| bf2ad7f1-3823-4ad2-a788-44a25827c93e | cirros            | ACTIVE |
| b3a4368b-7368-45e5-bfe4-63f59d732c41 | ubuntu 14.04      | ACTIVE |
+--------------------------------------+-------------------+--------+----------+

##Write Packer Template## Okay, time to get busy. Let’s write a template for Packer to create an image list. We’ll need to gather some info first.

  • Get your keystone info by catting out your keystonerc file. For me, this was cat keystonerc_admin. Some info below has been changed to protect the innocent.
[rsmitty@localhost ~(keystone_admin)]$ cat keystonerc_admin
export OS_USERNAME=admin
export OS_TENANT_NAME=admin
export OS_PASSWORD=testpass
export OS_AUTH_URL=http://192.168.1.200:5000/v2.0/
export OS_REGION_NAME=RegionOne
export PS1='[\u@\h \W(keystone_admin)]\$ '
  • Create a new json file somewhere on your machine. I simply called mine packer_template.

  • There’s a lot of options for Openstack in Packer (found here). Some of this will vary by the way your particular Openstack deployment is set up, but for me, this template contains all of the necessary basic fields:

{
  "builders": [
    {
      "type": "openstack",
      "username": "admin",
      "password": "testpass",
      "provider": "http://192.168.1.200:5000/v2.0",
      "ssh_username": "ubuntu",
      "project": "admin",
      "region": "RegionOne",
      "image_name": "Packer Test Image",
      "source_image": "b3a4368b-7368-45e5-bfe4-63f59d732c41",
      "flavor": "0d7e469c-e99b-4267-b154-35874b224f54",
      "networks": ["0296eb7d-7f94-4cc1-b42f-f2d680b81359"],
      "use_floating_ip": true
    }
  ]
}

Notes about what’s what:

  1. username & password: Map to OS_USERNAME and OS_PASSWORD from source file
  2. provider: Maps to OS_AUTH_URL
  3. region: Maps to OS_REGION_NAME
  4. source image: UUID of the Ubuntu image we talked about earlier
  5. flavor: UUID of my m1.tiny flavor. Beware, this changes on any flavor update!
  6. networks: UUID of my private network. Can be an array of several networks.
  7. use_floating_ip: As mentioned earlier, floating IP allows Packer to actually SSH to this server across my home network.

##Test Time!## Let’s see if this thing will actually create an image for us.

  • Save your template if you haven’t already.

  • Validate the template to make sure there aren’t any glaring errors with packer validate NAME_OF_TEMPLATE.json. This should return the text ‘Template validated successfully.’

  • Run the template with packer build NAME_OF_TEMPLATE.json. For me, this gave the following output when everything completely worked:

Spencers-MBP:Desktop spencer$ packer build packer_template
openstack output will be in this color.

==> openstack: Creating temporary keypair for this instance...
==> openstack: Waiting for server (82db25b2-e1a5-4aef-be4a-cfccf744e103) to become ready...
==> openstack: Created temporary floating IP 192.168.1.204...
==> openstack: Added floating IP 192.168.1.204 to instance...
==> openstack: Waiting for SSH to become available...
==> openstack: Connected to SSH!
==> openstack: Creating the image: Packer Test Image
==> openstack: Image: 70a610e9-302a-40f4-a4ca-59b6ad260e63
==> openstack: Waiting for image to become ready...
==> openstack: Terminating the source server...
==> openstack: Deleting temporary keypair...
Build 'openstack' finished.
  • Nice! Seemed to work. Now if we head out to the Glance UI, we can see that our shiny new image hanging out!

##Well, Now What?## So we’ve built an image with Packer, which is great. But the real value here comes with building on multiple platforms at the same time and also doing some provisioning to install the necessities before creating the image.

This tutorial is getting pretty long in the tooth, so I’m not going to add another provider to create an image on, but I do want to actually install something to actually change something about the image. Let’s install Apache as part of the build. Note that in a proper environment, we would probably just install Apache and we would let our config manangement tool handle deploying our webpage, since that’s the kind of thing we would want to checkout from version control at boot time.

Here’s the template:

{
  "builders": [
    {
      "type": "openstack",
      "username": "admin",
      "password": "testpass",
      "provider": "http://192.168.1.200:5000/v2.0",
      "ssh_username": "ubuntu",
      "project": "admin",
      "region": "RegionOne",
      "image_name": "Packer Test Image",
      "source_image": "b3a4368b-7368-45e5-bfe4-63f59d732c41",
      "flavor": "0d7e469c-e99b-4267-b154-35874b224f54",
      "networks": ["0296eb7d-7f94-4cc1-b42f-f2d680b81359"],
      "use_floating_ip": true
    }
  ],
   "provisioners": [{
    "type": "shell",
    "inline": [
      "sleep 30",
      "sudo apt-get update",
      "sudo apt-get install -y apache2"
    ]
  }]
}

So, I ran into a need to scan some files for viruses on Ubuntu this past week. However, a couple of things prevented this from being straight forward:

  • I couldn’t be sure where these files were being stored.

  • I didn’t want to scan the entire filesystem just to get at these few files.

So I went out looking for a way to scan only new files with clamscan. After a good bit of digging, I ran across this old thread where someone had a similar question. So after getting a pointer there, I was able to make this happen. Here is the way to do it:

##Install ClamAV##

  • Issue sudo apt-get install clamav to get the freshclam and clamscan commands.

  • Update the virus definitions with a sudo freshclam. This will take a few seconds the first time.

##Get Ya Find Right##

Now this took a bit of playing around with to get where I wanted. I wanted to find all files of a certain type that had been created or modified in the past week. There’s also some differences to note about mtime vs. ctime vs. atime as a flag for find. Linux-FAQs.info did a good job of explaining these differences:

ctime - ctime is the inode or file change time. The ctime gets updated when the file attributes are changed, like changing the owner, changing the permission or moving the file to an other filesystem but will also be updated when you modify a file.

mtime - mtime is the file modify time. The mtime gets updated when you modify a file. Whenever you update content of a file or save a file the mtime gets updated.

*Most of the times ctime and mtime will be the same, unless only the file attributes are updated. In that case only the ctime gets updated.*

atime - atime is the file access time. The atime gets updated when you open a file but also when a file is used for other operations like grep, sort, cat, head, tail and so on.

For my purposes, I wanted EVERYTHING that had changed, so that pointed to using the ctime flag. For a first test, I just wanted to see how many items find would return. I was able to do that by issuing (edited to look for .md files, just for laughs): sudo find / -name "*.md" -ctime -7 -type f | wc -l. That command simply returns the number 5 on my machine at the time of writing. Let’s not pipe it out to wc so we can see those files:

rsmitty@rsmitty-notebook:~$ sudo find / -name "*.md" -ctime -7 -type f
/home/rsmitty/Desktop/usenet-dashboard/README.md
/home/rsmitty/Desktop/rsmitty.github.io/_posts/2014-09-07-creating-a-blog-2.md
/home/rsmitty/Desktop/rsmitty.github.io/_posts/2014-09-16-clamscan-files-by-date.md
/home/rsmitty/.xbmc/addons/service.xbmc.versioncheck/README.md
/usr/share/xbmc/addons/service.xbmc.versioncheck/README.md

Note: You’ll have to do almost everything with ClamAV as root user. Also, you can change the number of days to scan for by changing the ‘-7’ portion of the command.

Okay, slightly more interesting, and now we know what we’re working with. Let’s move on…

##Enter Xargs##

Using xargs was a first for me as part of this little endeavor. It’s a really handy tool to add to the toolbox! If you’ve ever tried to run a bash command like rm * and received an error like “Argument list too long”, xargs is the answer to your problems. It takes the argument list and breaks it down into smaller pieces. It’ll then run subsequent commands with each sublist. For the purposes of what I was doing initally, there were about 7,000 files to scan, xargs was able to break those up into two scans of ~3,000 files and one scan of ~1,000. Worked great!

Here’s the final command that I used:

sudo find / -name "*.md" -ctime -7 -type f -print0 | sudo xargs -0 clamscan --remove --log=/home/rsmitty/clamscan.log
  • The –remove flag just means that if a vulnerability is found in that file, delete the file immediately.
  • the –log flag sets the path of the log file that clamscan will write. You will need to do this for sure if you have lots of files to scan, because several scan summaries will be written to this file.

After running, the log file will look something like this:

rsmitty@rsmitty-notebook:~$ sudo cat /home/rsmitty/clamscan.log 

----------- SCAN SUMMARY -----------
Known viruses: 3560111
Engine version: 0.98.1
Scanned directories: 0
Scanned files: 5
Infected files: 0
Data scanned: 0.02 MB
Data read: 0.01 MB (ratio 2.50:1)
Time: 6.441 sec (0 m 6 s)

After part one of this tutorial, we have a development environment setup that’s ready for us to really start making some headway in getting a presentable blog up and running. In this tutorial, I’ll show how I plugged Bootstrap’s example blog theme into my site and also guide through creating a first post.

###Create A Default Layout

  • First, we’ll want to get our code syntax highligting to work. Create a CSS file in the root of your Github repo called pygments.css by issuing: pygmentize -S default -f html > pygments.css

Note: This step may not be necessary for everyone, but I’ll be including a call to pygments.css in the next step. So if you’re following along step-by-step, it may be worth going ahead and doing this.

  • Let’s create a default template for all of our pages to use. This is where the special folders that we created in the last tutorial come into play. Change into the _layouts directory and open a file called default.html for editing. I’ve created a basic template that imports the necessary CSS and Javascript files for the Bootstrap example blog. I’m relying on the public URLs for those imports; however, if you want to ensure that they are always functional, you’ll want to download these files manually and import them from a path inside your repo. Add the following to the default.html file:
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta content='text/html; charset=utf-8' http-equiv='Content-Type'>
    <meta http-equiv='X-UA-Compatible' content='IE=edge'>
    <meta name='viewport' content='width=device-width, initial-scale=1.0, maximum-scale=1.0'>

    <title>{{ site.name }}</title>

    <meta name="author" content="{{ site.name }}" />
    <meta name="description" content="{{ site.description }}">

    <!-- Bootstrap CSS Files -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css">
    <link rel="stylesheet" href="http://getbootstrap.com/examples/blog/blog.css">
    
    <!-- Pygments CSS File for Syntax Highlighting -->
    <link rel="stylesheet" href="/pygments.css">

  </head>

  <body>

  <div class="blog-masthead navbar-fixed-top">
      <div class="container">
        <nav class="blog-nav">
          <a class="blog-nav-item" href="#">Home</a>
          <a class="blog-nav-item" href="#">About</a>
        </nav>
      </div>
    </div>

    <div class="container">
      <div class="blog-header">
        <h1 class="blog-title">{{site.name}}</h1>
        <p class="lead blog-description">{{site.description}}</p>
      </div>

    <div class="row">
      <div class="col-sm-8 blog-main">
          {{ content }}
      </div>

    <div class="col-sm-3 col-sm-offset-1 blog-sidebar">
      <div class="sidebar-module sidebar-module-inset">
        <h4>About</h4>
        <p>{{site.about}}</p>
      </div>
      <div class="sidebar-module sidebar-module-inset">
        <h4>Contact Info</h4>
        <ol class="list-unstyled">
          <li><a href="#" target="_blank">Github</a></li>
          <li><a href="#" target="_blank">LinkedIn</a></li>
        </ol>
      </div>
    </div>
  </div>
  </div>

  <!-- Bootstrap JS Files -->
  <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
  <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/js/bootstrap.min.js"></script>

  </body>
</html>

###Edit The Index Page

  • Let’s tell our index.html page to use our snazzy new layout. Edit your index.html page to look like the following:
---
layout: default
---

Hello, World!

Note: The YAML at the top of this page allows us to specify certain behavior from page to page. For example, we’re telling this page to load our default template. However, if we created another template called foo later on in the future, we could simply change the layout variable to point to the new foo layout.

  • Refresh the localhost:4000 page in your browser and you should see the new layout load, along with all of the site info that we entered in _config.yml in the last tutorial.

  • Looks good! Now, we’d rather the index page say something other than “Hello, World”. In fact, we want the first thing that people see to be a list of all of the highly interesting and potentially life changing posts that we’ll be making as our blogging continues. Let’s now edit our index.html to look like the following:

---
layout: default
---

{% for post in site.posts %}
    <div class="blog-post">
      <h2 class="blog-post-title"><a href="">{{ post.title }}</a></h2>
      <p class="blog-post-meta">Written {{ post.date | date: "%m.%d.%Y" }} by {{post.author}}</p>
        {{ post.content | truncatewords:40}}  
      <a href="" class="read-more">Read More</a>
      <hr>
    </div>
{% endfor %}

This tells Jekyll to run a for-loop before displaying the index.html page. As this loop runs, a div is created for each blog post, along with some info about the post (author, date, etc.). It also truncates the post to 40 words and provides a link to read the full text. A refresh will be pretty blank right now, as there’s not any posts and it looks a bit silly. Let’s create a dummy post to fill things out a bit.

###Create A Post

  • Creating a post is a bit of pretty cool black magic on Jekyll’s side. It expects a specific type of filename and reside in the _posts directory. So let’s create a file named like YEAR-MONTH-DAY-TITLE.md. At the time of this writing, I’m issuing touch 2014-09-07-hello-post-world.md.

  • Inside that file, we’ll add some YAML configuration to the top to tell it some info about layout, title, and author. Edit the file to look like this and feel free to the author and text to whatever you wish:

---
layout: posts
author: Spencer
title: Hello, Post World!
---

Here's some pretty incredible text to prove that it actually shows up in our index page and I'll continue rambling here in hopes that I hit forty words and the post gets truncated. That will allow us to see the "Read More" link appear on our post.
  • Refresh the localhost:4000 page and you’ll see your first post appear!

Nice! Until you click the “Read More” or the title of the post. There’s no posts.html in _layouts, which is what we specified at the top of our file just a second ago. We can’t really use the default layout we created, because it’s missing a couple of things like post title and a back button that we’d want on single posts, but not on the main page.

###Create A Layout For Posts

  • Create a file called posts.html and place it in the _layouts directory. Populate the file with the following:
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta content='text/html; charset=utf-8' http-equiv='Content-Type'>
    <meta http-equiv='X-UA-Compatible' content='IE=edge'>
    <meta name='viewport' content='width=device-width, initial-scale=1.0, maximum-scale=1.0'>

    <title>{{ site.name }}</title>

    <meta name="author" content="{{ site.name }}" />
    <meta name="description" content="{{ site.description }}">

    <!-- Latest compiled and minified CSS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css">

    <!-- Optional theme -->
    <link rel="stylesheet" href="http://getbootstrap.com/examples/blog/blog.css">
    <link rel="stylesheet" href="/pygments.css">
  </head>

  <body>

  <div class="blog-masthead navbar-fixed-top">
    <div class="container">
      <nav class="blog-nav">
        <a class="blog-nav-item" href="/">Home</a>
        <a class="blog-nav-item" href="#">About</a>
      </nav>
    </div>
  </div>

  <div class="container">
    <div class="blog-header">
      <h1 class="blog-title">{{site.name}}</h1>
      <p class="lead blog-description">{{site.description}}</p>
    </div>

    <div class="row">
      <div class="col-sm-8 blog-main">
          <div class="blog-post">   
            <h2 class="blog-post-title">{{ page.title }}</h2>
            <p class="blog-post-meta">Written {{ page.date | date: "%m.%d.%Y" }} by {{page.author}}</p>
            <hr>
              {{content}}
            <hr>
            <a href="/"><< Back</a>
          </div>
      </div>

      <div class="col-sm-3 col-sm-offset-1 blog-sidebar">
        <div class="sidebar-module sidebar-module-inset">
          <h4>About</h4>
          <p>{{site.about}}</p>
        </div>
        <div class="sidebar-module sidebar-module-inset">
          <h4>Contact Info</h4>
          <ol class="list-unstyled">
            <li><a href="#" target="_blank">Github</a></li>
            <li><a href="#" target="_blank">LinkedIn</a></li>
          </ol>
        </div>
      </div>

    </div>
  </div>

  <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
  <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/js/bootstrap.min.js"></script>


  </body>
</html>
  • Now, let’s click on a link to our post and see it in all of it’s glory!

###Closing Well, that’s pretty much it for how I created my new blog. Now is probably a good time to push all of this to Github and ensure it works properly if you’re following along with these instructions. I hope this has been a helpful write-up and I’m sure I’ll document other cool stuff that I run into as I learn to use Jekyll more effectively. It’s now up to you to create lots and lots of posts on your own blog!

As I wrote previously, I wanted to learn to use Github Pages to build a new blog and start keeping track of my personal projects and ideas. However, in getting started, I noticed that there really wasn’t a straightforward way to move from Github’s quick and dirty tutorial to actually creating a useful blog. There’s some tools like Jekyll Bootstrap, but it seems like it’s been a while since it has been updated. So after combing through a lot of other people’s pages and some Jekyll tutorials, I was finally able to get something presentable up and running. I figured I’d share the overall process I used as a first series of posts, in case someone else finds it useful.

This first post will center around getting a development environment setup and creating a “Hello, World!” type page to demonstrate the basics. So without further ado…

This guide assumes a standard Ubuntu installation and a pre-existing Github account.

###Create Our Github Page

  • First, we’ll want to create an empty repo out on Github. The naming here is quite important, so for a personal blog, you’ll want to name it like GITHUB_USERNAME.github.io. So login to Github and click “+ New Repository”. My repo info looked like this:

  • Now, let’s clone our repo locally. This should be as simple as a git clone https://github.com/USERNAME/USERNAME.github.io.git in some place that you’re comfortable with. If you change into that directory, you’ll see it’s currently empty, unless you created a README or LICENSE file as part of your repo creation.

Note: If you don’t have git installed, issue sudo apt-get install git to install.

###Install The Basics

  • Let’s grab some packages we need to serve our site locally. This will be useful when writing posts, so you don’t have to push to Github constantly just to see what it looks like. Issue sudo apt-get install ruby ruby1.9.1-dev python-pip.

  • We’ll need a Javascript parser and the Jekyll gem, so get NodeJS and Jekyll from rubygems: sudo gem install node jekyll

  • This step is optional for some, but I needed syntax highlighting for code blocks. So install the Pygments package for Python: sudo pip install Pygments

###Create Our Folder Structure

There’s a certain folder structure that Jekyll looks for in your repo. For example, all posts must live in the ‘_posts’ folder and must be titled like YEAR-MONTH-DAY-TITLE.FORMAT. To help understand, the title of this post is 2014-09-06-creating-a-blog-1.md. Anyways, I wanted to get up and running with the absolute basic amount of stuff needed in my repo so here’s what seemed to work.

  • Change into your repo directory in a terminal. Create two directories, _layouts and _posts. They must be named exactly that, including the underscore at the front. Issue mkdir _layouts _posts

  • Let’s also touch a couple of files that we’ll use later on. Issue touch _config.yml index.html

###Configure The Site

  • We could probably do this part much later in the tutorial, but let’s go ahead and put some basic info about our site in the _config.yml file. Feel free to edit these fields to your liking, with the exception of the ‘highlighter’ option. That is for our code highlighting.
name: "Test Blog"

description: "A snazzy new blog, just for me!"

about: "Here's a bit of text that we'll put in an about section later on in our tutorial. I'm intentionally making it a little longer so that the section will look fuller later..."

highlighter: pygments

###Test Our Local Server

  • Now, what’s a tutorial without a “Hello, World!” portion? Let’s go ahead and edit index.html to say that. The “Hello, World!” text should be the only thing in the file.

  • In the root of your repo, let’s tell Jekyll to server our pages so we can see if everything works. Issue jekyll serve --watch. The watch flag will monitor the repo for changed files and regenerate automatically, so you don’t have to start/stop the process. However, I’ve noticed that’s not the case for certain files, namely _config.yml. Your terminal should look something like this when you start the process:

Configuration file: /home/rsmitty/Desktop/rsmitty.github.io/_config.yml
            Source: /home/rsmitty/Desktop/rsmitty.github.io
       Destination: /home/rsmitty/Desktop/rsmitty.github.io/_site
      Generating...
                    done.
 Auto-regeneration: enabled for '/home/rsmitty/Desktop/rsmitty.github.io'
Configuration file: /home/rsmitty/Desktop/rsmitty.github.io/_config.yml
    Server address: http://0.0.0.0:4000/
  Server running... press ctrl-c to stop.
  • Open the address http://localhost:4000 in a browser and you should see your “Hello, World!” text. Neat!

###Check It Out On Github

Okay, so we’ve got our first official page. How do we get that out to Github? Turns out, it’s really easy, because we’ve already got everything laid out just right locally.

  • Add all of our files to be commited: git add --all

  • Commit them with a helpful message: git commit -m "Committing basic folder structure and Hello World page."

  • Push it to Github: git push origin master

  • Give it a few seconds, and you should see your new page at your USERNAME.github.io address:

###Closing That’s it for part one of this tutorial. We’ve now got a development environment setup and know how to push our pages to Github. In part two, we’ll explore how to use Bootstrap to setup a blog layout and create some custom layout pages so that our site looks clean and user friendly.

Welp, after a long time away from anything blog related, I’ve decided to spin up a new site to start trying to keep better documentation of the stuff I do personally. A friend of mine asked about my old blog last week and, after telling me that he had liked to read it, I decided that maybe now was as good a time as any to get back into the swing of things. Life has changed quite a bit since the old entries I had made on my previous site, so the focus of my write-ups here will be pretty drastically different.

I had originally stopped writing mainly because, as graduation neared at Clemson, I was short on time and frantically trying to find a job to transition to ASAP. I also didn’t want to pay yearly for a domain name and to host my site somewhere. So in getting back into this, I have made the transition to Github Pages as a free spot to host my stuff. Pages uses the Jekyll framework to host static sites, which sounded like a neat platform to try out. This has also given me a chance to use git in a different way and learn the markdown syntax. I hope that I’ll be able to expand on these skills more in the future.

As for most of my new posts, they’ll probably center around the creation of this site (it took me a while to figure out the simplest path forward for Jekyll + Bootstrap), Linux system administration, OpenStack oddities and how-tos, and automation tools like Chef and Ansible. Finally, in regards to my older stuff, I was able to dig the old site out of the archive.org Wayback Machine (hooray for not keeping backups), so I’ve ported a few of the ones that were still somewhat interesting to this site.

Hope you like it!