Technical Community - Background Reading


The Chat topic for this week isthe Dell Crowbar Cloud open source project and DevOps with special guest Rob Hirschfeld.

Crowbar is an open source cloud deployment framework originally developed by Dell to support our OpenStack and Hadoop powered solutions. Recently, it’s scope has increased to include a DevOps operations model and other deployments for additional cloud applications.

https://github.com/dellcloudedge/crowbar

http://en.community.dell.com/dell-blogs/enterprise/b/tech-center/archive/2011/10/18/dell-crowbar-project-expanding-the-community-a-fall-2011-update.aspx

http://en.community.dell.com/dell-blogs/enterprise/b/inside-enterprise-it/archive/2011/08/17/crowbar-a-tool-for-speeding-up-open-source-cloud-installations.aspx

http://robhirschfeld.com/2011/07/28/crowbar-brings-devops-to-openstack-cloud-no-soup-for-you/ 

http://mdzlog.alcor.net/2010/06/08/devops-and-cloud/

Chat Transcript

Dell-PeterT Hi everyone
Lee4Dell Hey Peter!
Dell-LanceB Hello there ...
Dell-PeterT Rob and I are sitting in a room ready to get started in a second, as soon as he is able to log in
Dell-PeterT Hi Dammmifino and Dennis
Dell-PeterT Hi It guru
Dell-DennisS hi
Dell-PeterT we're getting Rob logged on
dammmifino afternoon
Dell-PeterT we'll probably start in a few minutes
Dell-PeterT Rob is having some technical difficulties with his browser, so he's going to start chatting using my account
Dell-PeterT Just as an introduction, Rob joined Dell a couple of years ago to help found the Cloud solutions group at Dell
Dell-PeterT He's here today to talk about recent developments and answer questions about Crowbar, the open source project and to answer general questions about cloud and Dell
Dell-PeterT so if you have any questions about Hadoop, Openstack, and Devops, feel free to ask
Dell-PeterT for all of you guys joining just now, you can catch up on the chat by clicking Action > Recent Room History
Dell-PeterT Is there any particular subject you guys want to learn about before we get started?
nateK Could Rob describe his overall experiences with Openstack
dammmifino definitely interestedin Hadoop but Dell direction around teh cloud as well...
Dell-PeterT Hi Sam
samantha_needham Hi
Dell-PeterT alright with that I'll let Rob take the floor from my account - he's going to take over from here
Dell-PeterT Hello!
Dell-PeterT (rob) I've been involved in Openstack since before it went public as a project
Dell-PeterT (rob) Frankly, it's been surprising how quickly the project gained traction
Dell-PeterT (rob) We expected it would be an important platform, but the level of interest has exceeded our expectations
Dell-PeterT (rob) it's causing the community around the code base to grow up very fast
Dell-PeterT (rob) While we've seen a lot of participation, there are some places where it looked like it would be more "even"
hansdeleenheer Belgian ::beer for everyone - Hello!
Dell-PeterT (rob) For example, the early indiciations were that we'd be on every hypersor equally
Dell-PeterT (rob) but there are some platforms (like Kvm) that have more traction based on where the developers are working most
Dell-PeterT (rob) - moving off Openstack for a minute)
Dell-PeterT Dammminino - do you have a specific question about Hadoop?
nateK Rob: As someone who spent the last 4 months working with Eucalyptus and encountered numerous issues, I recently switched over to using Openstack and have been quite impressed. Would you say Openstack is becoming the de-facto open source cloud computing platform? Or at least the one with the most community momentum?
Dell-PeterT Today we announced that we were going to open source the Hadoop barclamps
Dell-PeterT sorry.. alll Dell-petert is Rob
Dell-PeterT If the people need, I can talk about barclamps and Crowbar architecture...
Dell-PeterT Openstack has many more distributed contributors than Eucalyptus
phaseshifter is there a Gui to go along with openstack for deploying Vms?
Dell-PeterT that means that there is more of a diverse community. Even at it's roots, Openstack started from two different sources
Dell-PeterT also there is no "premium" version at the core of the offering
phaseshifter also, is the dell / openstack solution being "certified" on the C series gear or can it be purchased using large dual / quad socket boxes with 6 and 10 core processors and huge amounts of ram?
nateK I have a Loe question when it comes to Crowbar/barclamps: I've got a 4 machine cloud testbed...am trying to determine the Loe it would entail to learn Crowbar/chef/barclamps for such a small setup...? Your thoughts? On what size installations are you most often using Crowbar? Would you consider using it even for a setup of our size?
Dell-PeterT phaseshifter: right now, our certification process is "C" series gear
Dell-PeterT that's if for the Bios & Raid configurations
Dell-PeterT however, Crowbar is not hardware restricted
Dell-PeterT the Openstack installation that we are working on could be deployed on any systems that respond to a Pxe boot
Dell-PeterT you would just have to do the Bios and Raid configuration manually
phaseshifter Petert: thanks for the info.
phaseshifter I'm currently investigating both cloudstack and openstack
Dell-PeterT our roadmap includes expanding the certification to include "R" series gear
Dell-PeterT matek: Loe = Level of Effort.
Dell-PeterT matek: no, we tried hard to make it easy to get started
Dell-PeterT Our objective for Crowbar is that it provides a very fast initial install for Poc and labs
Dell-PeterT moving into a production environment will require deeper learning
Dell-PeterT but we did not want everyone to have to learn all the ins and outs of Openstack just to start using it
Dell-PeterT that was very important to us because learning to configuration is a big barrier to a lab install
Dell-PeterT it can also be error prone
Dell-PeterT That was part of our motication for open sourcing the Hadoop barclamps
Dell-PeterT because that configuration is also very complex
Dell-PeterT Ok Rob is back in the chat as Dell-robh
nateK Rob: Thanks. I will look into Crowbar based on your Loe recommendations and the fact it was designed with small poc and labs in mind. :-)
Dell-RobH_ I'm now under my own alias!
Dell-RobH_ One of our initial experience with Cloud deployments Before Openstack
Dell-RobH_ was that you needed to be able to setup in a lab with all of the prereqs included
Dell-RobH_ that's why Crowbar includes all the Dns, Dhcp, Ntp, monitoring, and performance tools
Dell-RobH_ we don't want anyone to have to round up all the services to get started
Dell-RobH_ once you move to production, you can swap out those parts with data center class components
Dell-PeterT Hi Bruno, to catch up with the conversation, click on Action > View Recent History
Dell-RobH_ phaseshifter: the latest version of Openstack (called Diablo) includes a Gui called the dashboard (now project Horizon)
Dell-RobH_ we're including the Dashboard in our Openstack deployment
Dell-RobH_ because it makes it much easier to get started with Openstack when you can use the self service portal
Dell-RobH_ if you have automation that uses Ec2 apis, then those will work too
Dell-RobH_ I have a video posted that includes a tour of the Openstack dashboard
phaseshifter Robh: good to know. Thanks.
Dell-RobH_ pausing for questions...
Dell-RobH_ phaseshifter: what work loads are you planning with your larger core boxes?
Dell-PeterT If anyone needs to catch up on the conversation, make sure to click on Action > Recent Room History
Dell-PeterT Hello Josh
DellSanMan This is very exciting stuff!
josh_g hello
Bruno_Vieira is there any tool to calcule the Iops?
phaseshifter Robh: no so much huge workloads as the ability to get a lot of Vms onto a single box while keeping power / space footprint low
Dell-PeterT @josh if you want to see the questions that have been asked earlier, click on the action tab and view recent history
josh_g thank you
Dell-RobH_ we're finding that there's a balance between disk, compute, and network
Dell-RobH_ if you pack too many Vms onto a single machine then you can overwhelm the disk i/o and network
Dell-RobH_ so most of our customers are looking at 2 socket w/ 4 or 6 cores each
Dell-RobH_ total of 8 to 12 core
Dell-RobH_ if you are using a lot of San, then storage = network
phaseshifter yeah, many of our current boxes are 2 x 6 core w/ 192gb of ram
Dell-RobH_ it's one of the reasons that customers are still using local disk
Dell-RobH_ with a iscsi persistent disk for the Vms
Dell-RobH_ which is very much like the Ec2 model
phaseshifter we're looking to do an iscsi backend with 10gbe
Dell-RobH_ We have a model that is 4 socket (the C2145) that is a great compute heavy unit
Dell-RobH_ phaseshifter - we're seeing a drive to 10gbe too
Dell-RobH_ using 10gbe makes sense for a future proofing.
Dell-RobH_ but can get expensive when you add in redundant legs per node
Dell-RobH_ depending on your application's elasticity, you make be able to have less redundnacy
phaseshifter the problem with our environment is that this is going to be a "staging" environment, but basically needs to be up 24/7
phaseshifter so redundnant 10gbe iscsi links on each compute node will be required
Dell-RobH_ understood
Bruno_Vieira is there switch Force10 for edge tiering? to complete all tiering with this solution...
Dell-RobH_ phaseshifter: that's what we are seeing in a lot of our designs
Dell-RobH_ it's reasonable to start with 1g and plan to migrate
Dell-PeterT Hello Dave, welcome to the Dell cloud chat - to see what we have been talking about, click Action > View Room History
Dell-RobH_ especially for a lab system
Dell-RobH_ Bruno_vieira: there is a F10 top of rack (tor) option
Dell-RobH_ we are working with them on specifications, I cannot make recommendations about it at this time
Dell-RobH_ I can tell you that our current designs recommend using a Lag between Tor switches
Dell-RobH_ so there is a fast interconnect within a row
Dell-RobH_ that avoids routing traffic to the core when possibel
Dell-RobH_ Bruno: so yes, it would be possible to have an all F10 solution
phaseshifter is xenserver the preferred compute node Os for cloudstack?
Dell-RobH_ Yes
phaseshifter s/os/hypervisor/
Dell-RobH_ phaseshifter: yes, Cloudstack is focused on Xenserver
Dell-RobH_ it's Not exclusive to Xenserver
phaseshifter err, sorry, I meant to say openstack
Dell-RobH_ lol - do that all the time
Dell-RobH_ Openstack dev seems to be more focused on Kvm
Dell-RobH_ but Xen and Xenserver are also very active
Dell-RobH_ Hyperv is getting some attention too
phaseshifter does dell have any plans to do anything with cloudstack or is openstack the settled upon Iaas solution?
Dell-RobH_ It's important to understand the different architectures between Kvm and Xenserver and how Openstack uses them differently
phaseshifter can you elaborate on the differences?
Dell-RobH_ phaseshifter: I cannot comment about Cloudstack plans. My team is 100% focused on Openstack as our cloud deployment
Dell-RobH_ differences...
Dell-RobH_ Kvm is a hypervisor that runs as a service on an operating system
Dell-RobH_ Xenserver (& Esx) are locked - they are both the Os and Hyervisor
Dell-RobH_ There is a version of Xen that runs as a service too
Dell-RobH_ but it is not supported by Citrix
Dell-RobH_ Openstack operates by deploying agents that interface with each node in the system
Dell-RobH_ in a Kvm (or Hyperv or Xen) environment the agent talks directly to the service (using libvirt)
Dell-RobH_ if you have an integrated hypervisor (xenserver or Esx) then you have to access the hypervisor via Apis
Dell-RobH_ most of the development that we are tracking focuses on the agent on the node approach
Dell-RobH_ but I know there are developers working on the other appoach too
Dell-PeterT @dell-robh can you talk a little bit more about Devops and Dell's involvement?
Dell-RobH_ we're nearly out of time...get in last questions...i'll take ones over Peter's
Dell-RobH_ (sorry Peter)
Dell-PeterT it's ok. definitely should take customer questions first :)
Dell-RobH_ Devops is a new approach to managing data center operations that started in cloud environments
Dell-RobH_ there are several goals for Devops
Dell-RobH_ 1) continous delivery / deploy of applications
Dell-RobH_ 2) change applications to make them easier to deploy and more elastic
Dell-RobH_ 3) create automation for deployments so that you treat your infrastructure setup as a code
Dell-RobH_ For our largest customers (and I mean Huge data centers of >10,000 nodes), they approach their applications
Dell-RobH_ as integrated with the deployment
Dell-RobH_ and they are constantly deploying their applications
Dell-RobH_ so individual nodes in their infrastructure are really not important
Dell-RobH_ it makes them very resilliant and elastic
Dell-RobH_ So it saves them a lot of time
Dell-RobH_ and allows them to buy much more basic servers with simpler configurations
Dell-RobH_ However, there is an upfront investment to build the automation
Dell-RobH_ That;s a big part of what we're delivering w/ Crowbar
Dell-PeterT can you give everyone an idea of who these huge Dell customers are?
Dell-RobH_ it is basically a self assembling Devops environment
Dell-RobH_ behind the scenes, Crowbar is using Opscode Chef to do all the configuration management
Dell-RobH_ that's the same too that large customers are using and smaller companies setting up their applications in cloud hosting
Dell-RobH_ we really liked Chef and how it operates
Dell-RobH_ However, there were some gaps that needed to fill
Dell-RobH_ that's where Crowbar come in
Dell-PeterT @waitzman - welcome back - click action > recent room history to catch up
Dell-RobH_ Crowbar does bare metal configuration and operating system installation'
josh_g is there any work towards opensourcing cloud images? ubuntu seems to be well supported, but finding a centos cloud image to work with openstack is much harder
Dell-RobH_ Crowbar does network configuration
Dell-RobH_ Crowbar does system wide analysis and makes deployment recommendations
Dell-RobH_ Josh_g: I'm not sure - I thought there was a Centos image available
Dell-PeterT All we have 3-4 minutes left - if you have any last minute questions, ask now
Dell-PeterT Also, if you have any questions after the chat session today, you can find Rob at http://robhirschfeld.com or on Twitter @zehicle
Dell-RobH_ josh_g: My understanding was that you could import an Ami into Openstack
Dell-RobH_ we don't test that as part of our normal cycle
Dell-PeterT Alright great chat this week. Thanks to Rob for sharing lots of great information
Dell-PeterT Reminder to everyone on the chat that the Dell Techcenter website is migrating on 11.11.11 to a new site
Dell-PeterT so be prepared for a new look and feel to the site then
josh_g that would be interesting, I will look into that, thank you
Dell-PeterT also you can get in touch with all of us through Twitter as well on the About Us page on Dell Techcenter
Dell-PeterT @josh_g - the new site looks great. We hope that our customers will like the changes
Dell-PeterT thanks for joining the chat everyone, and remember to check out Rob's blog at robhirschfeld.com
Dell-PeterT have a good evening (or morning depending on where you are)
hansdeleenheer or night!
hansdeleenheer :)
Dell-RobH_ Thanks everyone!