Server Management & Source Control: The key to scalability and teamwork

System administrators are being asked to deploy more services with less help these days. With turnover also happening more quickly with the economy decline, its crucial that the IT staff has a well documented server configuration environment. The OSU Open Source Lab has utilized cfengine with an SCM for several years and was key to its growth and success. You should attend if you’re having trouble keeping track of configuration changes on all your servers, trouble keeping the team on the same page with those changes, or if you want to have a history of the changes you make.

The key to creating such an environment is a two fold process:

• Implementing a central configuration application (cfengine or puppet)
• Using source control management (SCM) with configuration files (git, svn, etc)

While there are several SCM’s to choose from out there, this session will focus primarily on using git in conjunction with cfengine. Any material and ideas from this session can easily be adapted to other systems such as puppet with subversion, etc. This also session assumes the attendee has experience with some form of SCM (preferably git), and cfengine. I will go over small details of each but the primary goal is to describe the work flow.

• Summary (why do this?)
  • OSL cfengine+scm success story
  • Learning tool for new comers (like students!)
  • Easy to see global-wide changes as a group
  • Encourages self document ion and group collaboration
• GIT
  • Small overview
  • Offline use (I made some commits at 30k ft and pushed later)
  • Branching is easy (great for major system changes)
  • Show helpful links to sites about git
  • Gives flexibility on how changes get pushed (everyone vs. merger)
• Cfengine
  • Quick overview of what cfengine does
  • Describe briefly how the OSL uses cfengine
  • Key Features
    • Easy deployment
    • Standardized
    • Central configuration
  • Group by service easily
  • Has its own language
• Integrating cfengine with git
  • Repository layout
    • cfengine/inputs
    • files/{etc,var}
    • All in the same git repository
  • Show the work flow
    • Clone git repository
    • Commit change & push change upstream
    • SSH keys sync git repo on gold server immediately
    • Run cfagent manually or wait for next run
  • Git repository setup
    • Central server with ssh access
    • Shared repository
  • SSH keys
    • Passwordless keys used to sync repository on gold server instantly
    • Show details on what to add, setup, etc
  • Email
    • Send all diff’s via an email hook
    • Key tool in watching changes happen in real-time
• Working on bigger changes
  • Branches
    • Want to implement a major change? Create a branch
    • Merge change (easily work though conflicts with git)
    • Nice for quick changes that you don’t want to push
• Secure repository
  • For files like SSL certificates
  • Secure jump host, no remote checkouts
  • Diffs? Use gpg encryption

Integrating revision control into a system like cfengine makes managing hundreds or thousands of machines a lot easier. Emails act like a self policing mechanism inside the group where each person and see the change in real time and suggest alternatives. As a whole in creates an environment where you have an excellent idea of how your servers are setup and everyone in your group also knows.