Datacenter tools?

Michael T. Halligan michael at halligan.org
Thu Jun 3 11:40:48 PDT 2004


My ops director has a good-sized chunk of cash in our budget
marked aside for "Datacenter tools".  I was wondering what people like
to keep on their datacenter for such things.

So far, we have : 

- Plantronics Industrial noise-canceling phono dual-ear headset,
  with connectors for the 3 different types of cell phones our ops
  people use, and one portable phone we keep in a cabinet.
- Power convertors for the 3 different types of cell phones
- anti-static straps in each cabinet
- assorted sizes Velcro cable ties, as well as nylon zip ties
- 1 APC Environmental Monitoring Unit in each cabinet
- 1 Laptop in each cabinet with serial connections and adapters for
  3 different types of serial connections
- Ethereal & Minicom installed, as well as a web browser
- Patch panel for laptop with 3 network connections (one for each
  subnet)
- Assorted tools (punchdown, cable cutters, screwdrivers, wrenches (for
  adjusting cabinets), several paper-clips (APCs have very tiny reset
  buttons if they trip)
- Roll of quarters for vending machine
- One cold-spare for every classification of server, including load
  balancers, switches, firewalls
- 4U chatsworth locking drawers in every cabinet to keep spare parts and
- Cold spares for anything that we could imagine will break (CPU,
  memory, different sizes of hot-swap drives, raid cards, PDUs (very
  helpful, just had a PDU fail on us yesterday), GBICs,  we've even got a spare
  1U USR tray of fans in case one of ours fails. 
- Paper & pens
- 2 Label makers, with extra tape
- Post-its
- Multimeter
- Extra cables of all types (serial, scsi, cat5, cat5 crossover)
- Power tester
- Ethernet link checker
- Floor panel lifters
- Foldup nylon chair
- 2 Keyboards & Mice (less important since our datacenter provides
  rolling carts)

It sounds a bit cluttered, but we have 2 cabinets dedicated for testing
environment and management tools, and production in other cabinets,
eventually we'll have 2 cabinets solely for testing, and one for
management).

We've tried to make everything as remotely manageable a possible.
Everything sits on a remote serial console, as well as a remote
reboot switch. Our datacenter has a staffed 24x7 NOC that will change
tapes, sit down at a terminal to read errors and type commands, as well
as reboot boxes.

We've also made everything redundant as possible, having everything
that's not using our clustering utilities have a hot-spare (vrrp for
network gear, LVS for servers).  We've also setup automatic rebuild
scripts for all of our servers, so we can kickstart or systemimage
through a serial console.

My goal is to eliminate trips to the dc as much as possible, but making 
sure we have every tool we need in case of any unforseen failure.

Some of the proposed new tools have been :

 - 1 AXIS network camera per cabinet (the kind that pan, so we can
   potentially look at blinkys, as well as who's accessing)
 - Fluke network debuggers
 

Does anybody else have any other good ideas for tools to keep in the datacenter?








More information about the Baylisa mailing list