Archive for the ‘vmware’ tag

Unattended install of vFabric software on RHEL using yum.

with 2 comments

If you’ve touched a server in the last 5 years or so you probably know that automation matters a lot. There are a lot of things that might prevent full automation when you deploy software into production, and the situation tends to be especially bad for commercial software. Maybe it’s defective installers, maybe it’s manual EULA acceptance or clickthrough agreements, account logins that you might need to reset, or maybe it’s software that’s not designed to work out-of-the-box as soon as you install it. Thankfully most of this stuff is slowly and steadily dying out.

With vFabric one of our goals was to enable fully automated deploys in a way that fits into the broader context of tools you’re either already using or seriously considering.

For vFabric 5 we’ve focused our attention on enabling automation on RHEL. For now Windows users are a bit out of luck, partly because the automation capabilities on Windows are not as developed as they are on Linux. Look for this to improve in the future though.

Using yum to install vFabric software on RHEL.

Step 1 is to attach your RHEL system to our public repo. The easiest way to do this is to add a special RPM we have created that creates the appropriate files:

rpm -Uvh http://repo.vmware.com/pub/rhel5/vfabric/5/vfabric-5-repo-5-2.noarch.rpm

Alternatively you can create /etc/yum.repos.d/vfabric-5.repo with these contents:

name=VMware vFabric 5 packages - $basearch

You can also refer to the full documentation on the VMware web site. Once you’ve done this if you “yum search vfabric” you will get this listing:

All the various vFabric software is there and ready to be installed. Let’s install vfabric-gemfire and see what happens.

I get a summary of what will be installed including vfabric-gemfire and vfabric-eula. Let’s answer y to this question (equivalently we could have said yum -y) and see what’s next.

This is the standard vFabric EULA agreement you would click through on the website or if you ran the GemFire JAR installer. One option we have is to scroll to the bottom of this EULA, accept it, and be on our way. One other thing to note is that the EULA is only accepted once per OS, so if we install some more vFabric stuff we don’t have to go through this again.

Obviously, though, we can’t automate this. How can we do a true unattended install?

Doing a true unattended install on RHEL with yum.

We put a mechanism in place that allows the EULA to be “pre-accepted” through use of a particularly worded file. Here’s how it works.

If you put the text string I_ACCEPT_EULA_LOCATED_AT=http://www.vmware.com/download/eula/vfabric_app-platform_eula.html into a file called /etc/vmware/vfabric/accept-vfabric-eula.txt you will not be manually prompted to accept the EULA and vfabric software will install unattended. (You really should read the EULA we worked very hard on it.)

One of many ways to do that is as follows:

Here’s what a GemFire install looks like after you’ve done that:

No prompting. It’s pretty easy to see how you could integrate the entire process into a Chef recipe or a Puppet script, or even into more traditional software management systems. The repo is also set up so you can host it internally using Spacewalk, for example.

Other reasons to do yum installs.

There are a few other reasons to do yum installs of vFabric software, particularly when going into production. For instance, necessary user accounts will be created, init.d scripts will be created and so forth. Let’s take GemFire as the example again, if you installed it via its JAR installer all this work needs to be figured out by the end user. At this point in the evolution of IT operations the way init scripts should work and the way user accounts should be set up is pretty well baked, it’s not an area that users should be trying to innovate in, and it’s better to have all this stuff “just work” out-of-the-box when you install.

Written by Carter Shanklin

November 2nd, 2011 at 4:24 pm

Posted in vfabric

Tagged with , , ,

Find out which vCenter manages an ESX host.

with one comment

Ran into this problem today and didn’t see any info about it online. ESX stores the IP address of the vCenter that manages it in the HostConnectInfo object.

If you connect to ESX you can call QueryHostConnectionInfo on the HostSystem object to find out which vCenter manages it.

Here’s PowerCLI code that will do that.

Get-VMHost |
    Get-View -property Name |
    Foreach { $_.QueryHostConnectionInfo() } |
    Select ServerIp

Written by Carter Shanklin

May 24th, 2011 at 11:24 pm

Posted in Uncategorized

Tagged with , , ,

Vote For My Elastic Memory for Java VMworld Session Or Else!!

without comments

I tried not to get a swelled head when Stu Radnidge conferred divine status upon me some years ago. After all Stu grew up in the Australian wilderness and I suppose riding a kangaroo to school every day could lead to a warped sense of reality.

So it was refreshingly deflating to find that nobody even noticed my VMworld session (2917 vote for me). In fact permit me to helpfully reproduce what you would have seen if you had voted for me as you should.

Elastic Memory for Java is a project I’ve been helping out with for the past year or so that aims to let you run Java apps on ESX without having to give them all their memory all the time. Java apps run great on ESX, so long as they have all the memory they want. If they don’t, things can go south really quick.

This is a big problem because Java likes to be lazy about cleaning up after itself, and prefers to hold on to every piece of trash and dead memory in sight until it nearly collapses under its own weight. It’s sort of like the software world’s answer to the Collyer brothers.

This behavior also explains why traditional ballooning in ESX doesn’t work well with Java. EM4J is different because it’s designed to work within Java and has intimate understanding of what’s happening in the Java app. If you wanna find out more, you gotta come to the session.

Why is VMware doing this you ask?

  • Efficiency is in.
  • We’re on the precipice of an app explosion. We’ve already seen it in consumer space, more special purpose apps that individually do less than monolithic apps of the past. This is infiltrating enterprise and when it does we need an order-of-magnitude or two more consolidation to cope with it.
  • Oh yeah and cloud. Cloud cloud cloud cloud cloud. There I said it.

P.S. If you’re interested in trying Elastic Memory for Java out, we’re running a select beta. Message me on twitter or email me at cshanklin at vmware com and let me know you’re out there.

Otherwise see you in Vegas!

Written by Carter Shanklin

May 12th, 2011 at 12:03 am

Posted in Uncategorized

Tagged with , ,

ESX Performance Counter Secrets Revealed

with one comment

Top Secret

Successful Succession?

The future of ESX’s COS is still a contentious subject of speculation. And with its future in doubt, so is the future of the ESX administrator’s most beloved friend and constant companion, esxtop. Esxtop’s heir apparent, resxtop, remains plagued by platform restrictions and unclear compatibility. Can it really be a viable replacement?

Interestingly, the PowerCLI team introduced their own way of getting esxtop data, however the official documentation is a bit light on detail.

These both appear to use some form of secret, undocumented API. It’s not clear why such a vitally important data source needs to be shrouded in mystery. Thankfully, the following cable was intercepted by a brave and intrepid soul whose name I have sworn to anonymity. It sheds substantial light on the nature of this hidden API, and will hopefully prove an invaluable reference.

The aforementioned PowerCLI team provided some rudimentary discovery mechanisms, but they may not dig deep enough. For example, LucD notes on his blog that he found only 392 counters on his system, whereas the cable we obtained identifies 425 counters. Is there some dark conspiracy at work here? Worse, many of the counters appear to give misleading information, seeming to need adjustment before they can be used. Hopefully the information captured in this cable will resolve these questions once and for all.

Text of The Intercepted Cable

Thursday, 14 February 2008, 12:21
C O N F I D E N T I A L SECTION 01 OF 06 VMWARE 002012
EO 12958 DECL: 01/25/2018
Classified By: COM: Thomas Lindboe: For reasons 1.4 b/d


We must not let these values fall into the wrong hands. The fate
of the free world is in the balance. Full details attached.

What We Found

Our intercepted cable revealed a treasure trove of information. Not only did it give us a full readout of all statistics, but their names, the way they correspond to the names shown in esxtop, and possible clues for how to adjust misbehaving counters. We rely on all of  you to expose this data to the light and help us unravel this mystery.

Written by Carter Shanklin

December 19th, 2010 at 8:43 pm

Posted in Uncategorized

Tagged with , ,

Spring into lame puns with Roo.

without comments

Roo LogoI’ve never been much of a Java developer, back in my developing days I spent my time in just about anything but Java, whether scripting or compiled languages. But with the recent pairing of VMware and Salesforce, and then the announcement that VMware and Google are BFFs, Java suddenly seems a lot more relevant than ever.

So I decided I needed to be a bit more Java savvy. One of the most common technologies you’ll encounter when developing is so-called Object Relational Mapping or ORM. The concept here is really simple, ORM lets you treat rows in a database like they were real live language objects, so if you fetch objects you’re really fetching database rows and if you update objects you’re really updating database rows. The concept is pretty simple. Somehow, though, the implementation is really nasty. And I mean really, REALLY nasty.

I set out to learn probably the most popular Java ORM, hibernate, using a hibernate tutorial. The tutorial is pretty decent if you overlook the fact that it doesn’t actually work and you need to search the jboss forums for fixes to the Maven POM file (what is that anyway?). Well then there was the fact that I couldn’t actually get it working. In truth it was probably my fault, I probably got one of the 17 steps wrong in the 30 page tutorial, or maybe I did them out of sequence or something. But I got close enough that I decided that if I absolutely, positively had to get it working I probably could.

After this experience I can’s say I’ve become much more of a Java fan. I started to take a look at some of the other technologies involved in building Java apps and my head began to spin. Do I use hibernate or should I use JPA? Do those things even do the same thing? Do I need to use JTA with either of them? Is struts a part of tiles? Other way around? Or are they competing technologies? There are at least 3 logging frameworks in Java that seem to have good traction, which one should I use? Is TestNG better than JUnit? Does it really matter? A funny note in the Hibernate tutorial is that Hibernate supports *BOTH* c3p0 and proxool as JDBC connection pools (What?), but you should positively under no circumstances use the default JDBC connector that ships with Hibernate because it’s a piece of crap.

I’m ready to fly the white flag, this stuff is just too much for me. Call me a bad developer but I just don’t have the patience for all this nonsense. They have a term for this, “yak shaving”. Reminds me of the bad old days of UNIX, when you were thrilled if “configure; make; make install” didn’t require you to edit source code.

On day 2 of my Java adventure I decided to figure out Roo, mostly because I couldn’t figure out what it was or why anyone would use it. And I guess somebody must think it’s pretty important because it’s what Ben Alex showed off at Google IO. (Ben has a love for expense reports that borders on unnatural but it’s still a pretty good demo.)

Roo is a “convention over configuration” tool, which is just a fancy way of saying that it makes a bunch of decisions you really shouldn’t care about and it makes them automatically. No more scratching your head figuring out if you should use JMX or JPA or JINI or JPMC or JWTF. As someone who doesn’t know or care what’s going on under the covers this is really attractive.

So I gave it a shot, not trying to follow Ben’s IO demo, but rather the online Pizza Shop demo. (I have it on good authority that Ben once tried to expense over $13,000 worth of pizzas, probably where he got the idea for the expense report app.)

project --topLevelPackage com.springsource.roo.pizzashop
persistence setup --provider HIBERNATE --database HYPERSONIC_IN_MEMORY
entity --class ~.domain.Topping --testAutomatically
field string --fieldName name --notNull --sizeMin 2
entity --class ~.domain.Base --testAutomatically
field string --fieldName name --notNull --sizeMin 2
entity --class ~.domain.Pizza --testAutomatically
field string --fieldName name --notNull --sizeMin 2
field number --fieldName price --type java.lang.Float
field set --fieldName toppings --element ~.domain.Topping
field reference --fieldName base --type ~.domain.Base
entity --class ~.domain.PizzaOrder --testAutomatically
field string --fieldName name --notNull --sizeMin 2
field string --fieldName address --sizeMax 30
field number --fieldName total --type java.lang.Float
field date --fieldName deliveryDate --type java.util.Date
field set --fieldName pizzas --element ~.domain.Pizza
controller all --package ~.web

With this file in hand, all you’ve got to do is pipe it into Roo and after a few minutes you’ve got a working application. If we break the script down, there are 5 major things:

  1. Create the application using “project”
  2. Instruct it to use Hibernate and the Hypersonic in-memory database.
  3. Create entities (these correspond to database tables).
  4. Create the fields within the entities, and associations between the entities.
  5. Add a web layer using “controller”.

It’s easy to feed this to Roo automatically by just putting it into a file called, say, pizza.roo and running:

$ /path/to/roo/roo.sh < pizza.roo

Roo does all the grunt work of dealing with Maven, which in turn downloads all your dependencies (this can take a while the first time you do it), and creates lots of stub code for you. After this I ended up with 458 files which I could compile into a web app. This included a fairly complex Maven pom.xml file which I could use to do other things, such as launch the application. To do that, just type this in the root of your project directory:

$ mvn tomcat:run

You should note that this wants to launch its own Tomcat and HSQLDB. If you have local copies of either of these apps, there will be conflicts. Since I did, I decided the path of least resistance was just to shut down my system services and let Maven handle it all. After that launches you can navigate to localhost with your web browser as follows:

Working Pizza Shop App

The working pizza shop application.

Success! A working (sample) app without a line of code! True I didn’t write code, but more importantly this approach seems to make data the primary concern, which I find very attractive since combining, processing and displaying data is the real value of any application.

In this example everything, including the database, runs in memory, you don’t need anything installed except Roo, Maven and your JDK. To make things a bit more interesting I decided to link this application to a MySQL database that I have running on my host. This proved to be surprisingly easy, with only one gotcha. I went back into my Roo shell and typed in:

persistence setup --provider HIBERNATE --database MYSQL --username carter

This almost worked, but I had to manually edit a field in a file called database.properties to read like:


Here “hibernate” is the name of my database in MySQL. Ordinarily you’d probably want to use things like usernames and passwords but I like to live dangerously. With that in place I launched my application again by typing “mvn tomcat:run”. I was pleasantly surprised to find that everything worked! I took a peek in MySQL and couldn’t believe my eyes, all the right tables had been created automatically! Working with Roo is a real pleasure compared to slugging it out by hand with tools like Hibernate.

Showing the schema in MySQL

Shockingly, things "just worked".

I’m really impressed with Roo and think it’s one of the most interesting things going on in the SpringSource portfolio right now.

Written by Carter Shanklin

May 22nd, 2010 at 3:07 am

Posted in Uncategorized

Tagged with , ,