$blogname

Archive for the ‘vfabric’ tag

GemFire DTD

without comments

If you’re writing GemFire cache.xml files you’ll probably want to refer to the GemFire DTD files for valid syntax. These are hosted on gemstone.com but don’t seem to appear in Google searches so I’m putting this blog in place to facilitate finding them.

GemFire 6.6 DTD on gemstone.com (latest version)
Ye Olde GemFire 6.0 DTD (consider an upgrade)

Written by Carter Shanklin

March 23rd, 2012 at 11:14 pm

Posted in GemFire,Uncategorized,vfabric

Tagged with ,

Unattended install of vFabric software on RHEL using yum.

with 2 comments

If you’ve touched a server in the last 5 years or so you probably know that automation matters a lot. There are a lot of things that might prevent full automation when you deploy software into production, and the situation tends to be especially bad for commercial software. Maybe it’s defective installers, maybe it’s manual EULA acceptance or clickthrough agreements, account logins that you might need to reset, or maybe it’s software that’s not designed to work out-of-the-box as soon as you install it. Thankfully most of this stuff is slowly and steadily dying out.

With vFabric one of our goals was to enable fully automated deploys in a way that fits into the broader context of tools you’re either already using or seriously considering.

For vFabric 5 we’ve focused our attention on enabling automation on RHEL. For now Windows users are a bit out of luck, partly because the automation capabilities on Windows are not as developed as they are on Linux. Look for this to improve in the future though.

Using yum to install vFabric software on RHEL.

Step 1 is to attach your RHEL system to our public repo. The easiest way to do this is to add a special RPM we have created that creates the appropriate files:

rpm -Uvh http://repo.vmware.com/pub/rhel5/vfabric/5/vfabric-5-repo-5-2.noarch.rpm

Alternatively you can create /etc/yum.repos.d/vfabric-5.repo with these contents:

name=VMware vFabric 5 packages - $basearch
baseurl=http://repo.vmware.com/pub/rhel5/vfabric/5/$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-VFABRIC-5

You can also refer to the full documentation on the VMware web site. Once you’ve done this if you “yum search vfabric” you will get this listing:

All the various vFabric software is there and ready to be installed. Let’s install vfabric-gemfire and see what happens.

I get a summary of what will be installed including vfabric-gemfire and vfabric-eula. Let’s answer y to this question (equivalently we could have said yum -y) and see what’s next.

This is the standard vFabric EULA agreement you would click through on the website or if you ran the GemFire JAR installer. One option we have is to scroll to the bottom of this EULA, accept it, and be on our way. One other thing to note is that the EULA is only accepted once per OS, so if we install some more vFabric stuff we don’t have to go through this again.

Obviously, though, we can’t automate this. How can we do a true unattended install?

Doing a true unattended install on RHEL with yum.

We put a mechanism in place that allows the EULA to be “pre-accepted” through use of a particularly worded file. Here’s how it works.

If you put the text string I_ACCEPT_EULA_LOCATED_AT=http://www.vmware.com/download/eula/vfabric_app-platform_eula.html into a file called /etc/vmware/vfabric/accept-vfabric-eula.txt you will not be manually prompted to accept the EULA and vfabric software will install unattended. (You really should read the EULA we worked very hard on it.)

One of many ways to do that is as follows:

Here’s what a GemFire install looks like after you’ve done that:

No prompting. It’s pretty easy to see how you could integrate the entire process into a Chef recipe or a Puppet script, or even into more traditional software management systems. The repo is also set up so you can host it internally using Spacewalk, for example.

Other reasons to do yum installs.

There are a few other reasons to do yum installs of vFabric software, particularly when going into production. For instance, necessary user accounts will be created, init.d scripts will be created and so forth. Let’s take GemFire as the example again, if you installed it via its JAR installer all this work needs to be figured out by the end user. At this point in the evolution of IT operations the way init scripts should work and the way user accounts should be set up is pretty well baked, it’s not an area that users should be trying to innovate in, and it’s better to have all this stuff “just work” out-of-the-box when you install.

Written by Carter Shanklin

November 2nd, 2011 at 4:24 pm

Posted in vfabric

Tagged with , , ,

vFabric GemFire — 20 demos in less than 5 minutes.

with 5 comments

vFabric GemFire is a Java-oriented in-memory distributed caching platform. To translate that into English, it’s a way of sharing and processing that’s extremely fast because the data is kept live in memory, and is also extremely scalable because GemFire applications can automatically discover and share data between peers over the network.

GemFire can run peer-to-peer or standalone. When you’re using it peer-to-peer you embed GemFire into your application. When the application launches, it discovers all its peers automatically via the network.

You run GemFire standalone through something called cacheserver. In the standalone case you launch one or a few cacheservers and then a large number of clients can connect to it. This mode resembles what you’d see if you used something like memcache or redis. This mode is good if you need to centralize data, write data to disk or if you want to perform continuous queries.

GemFire includes an extensive example set that gives you a good sense of what’s possible. Unfortunately I found the getting started guide a bit hard to follow. Part of it was just the usual Java BS: any time you’re asked to set environment variables to get software working you know you’re in for a rough ride. But beyond that there areĀ  various different piece parts you have to collect and configure before you can really get started, and the way to run the examples themselves is not very clear, the directory structure of the sample files seems inconsistent with the way the instructions tell you to run the samples. Maybe that was my error but I bet a lot of people would make the same error.

To try to help things a bit I wrote a script that tries to completely automate the demo experience. The script downloads all the necessary code and then presents a menu where you can select any demo you want to run. Some of the examples require a cacheserver, and in those cases the demo script will automatically start and stop one for you. There’s nothing to think about, just run the script and it should work (leave me a note if it doesn’t!)

Download the script or just copy and paste it from the box below. After you download, just run it and you’re on your way (see below for a note on system requirements).

#!/usr/bin/python
 
import string
import time
import urllib
import os.path
 
# If you want to see the commands as they execute, set this to 1.
verbose = 0
 
files = {
  "gemfire" : {
    "name" : "GemFire JAR",
    "file" : "gemfire-6.5.1.jar",
    "url"  : "http://dist.gemstone.com/maven/release/com/gemstone/gemfire/gemfire/6.5.1/",
    "sum"  : "f9b633da145a2afb27d2015ab76010818c0c2ee2"
  },
 
  "antlr" : {
    "name" : "ANTLR JAR",
    "file" : "antlr-3.1.3.jar",
    "url"  : "http://www.antlr.org/download/",
    "sum"  : "acd5253cf1eba7bdb133f14cd77b0ba2fd219f98"
  },
 
  "examples" : {
    "name" : "GemFire Examples",
    "file" : "GemFire_v6.5_SampleCode.zip",
    "url"  : "http://community.gemstone.com/download/attachments/6032137/",
    "sum"  : "6a1d6f5e4a9a4e1236caecac2935b5cb841bcb10"
  }
}
 
demos = {
 'BenchmarkAckConsumer' : {},
 'BenchmarkAckProducer' : {},
 'BenchmarkClient' : { "cacheServerArgs" : "start cache-xml-file=xml/BenchmarkServer.xml" },
 'ClientConsumer'  : { "cacheServerArgs" : "start cache-xml-file=xml/Server.xml" },
 'ClientWorker' : {},
 'CqClient'        : { "cacheServerArgs" : "start cache-xml-file=xml/CqServer.xml" },
 'DataEviction' : {},
 'DataExpiration' : {},
 'DataOverflow' : {},
 'DataPersistence' : {},
 'DeltaPropagationClientFeeder' :
        { "cacheServerArgs" : "start cache-xml-file=xml/DeltaServer.xml" },
 'DeltaPropagationClientReceiver' : {},
 'DeltaPropagationServer' : {},
 'DistributedLocking' : {},
 'DurableClient' : {},
 'DurableServer' : {},
 'FunctionExecutionPeer1' : {},
 'FunctionExecutionPeer2' : {},
 'I18nClient' : {},
 'MultiuserSecurityClient' : {},
 'MultiuserSecurityServer' : {},
 'MyArrayListResultCollector' : {},
 'PartitionedRegionVM1' : {},
 'PartitionedRegionVM2' : {},
 'PushConsumer' : {},
 'PushProducer' : {},
 'Querying' : {},
 'SecurityClient' : {},
 'SecurityServer' : {},
 'Transactions' : {},
}
 
def download(url, file):
        urllib.urlretrieve(url, file)
 
def downloadFile(fileInfo):
        # Check to see if the file exists.
        if os.path.isfile(fileInfo["file"]):
                return
 
        # Otherwise download it.
        url = fileInfo["url"] + fileInfo["file"]
        print "Downloading " + fileInfo["name"]
        download(url, fileInfo["file"])
 
def runDemo(demoName, args=""):
        baseCommand = "java -classpath classes:../gemfire-6.5.1.jar:../antlr-3.1.3.jar"
        fullCommand = baseCommand + " quickstart." + demoName + " " + args
 
        os.chdir("quickstart")
        # Some demos require a cache server. Start one up.
        cacheServerNeeded = 0
        if "cacheServerArgs" in demos[demoName]:
                cacheServerNeeded = 1
                print "Starting a cache server for this demo."
                cacheServerCommand = baseCommand + \
                    " com.gemstone.gemfire.internal.cache.CacheServerLauncher " + \
                    demos[demoName]["cacheServerArgs"]
                if verbose:
                        print "Starting cache server using: ", cacheServerCommand
                os.system(cacheServerCommand)
 
        # Start the demo.
        if verbose:
                print "Starting demo using: ", fullCommand
        os.system(fullCommand)
 
        # Stop the cache server if we started one.
        if cacheServerNeeded:
                print "Stopping cache server."
                cacheServerCommand = baseCommand + \
                    " com.gemstone.gemfire.internal.cache.CacheServerLauncher stop"
                os.system(cacheServerCommand)
                if verbose:
                        print "Stopping cache server using: ", cacheServerCommand
 
        os.chdir("..")
        demoName = raw_input("Press enter to return to the menu.")
 
def doMenu():
        while 1:
                print "Ready to run a demo."
 
                # Present choices.
                after = ""
                demoNames = demos.keys()
                demoNames.sort()
                for demo in demoNames:
                        formattedDemo = "%-33s" % demo
                        print formattedDemo, after,
                        if after == "": after = "\n"
                        else: after = ""
                print
                input = raw_input("Enter your choice: ")
                tokens = input.split(" ")
                demoName = tokens[0]
                arguments = string.join(tokens[1:], " ")
                try:
                        demos[demoName]
                        runDemo(demoName, arguments)
                except KeyError, ValueError:
                        print "Unknown demo, try again."
                        time.sleep(1)
 
def main():
        # Ensure we have all the files we need.
        for file in files.keys():
                downloadFile(files[file])
 
        # Extract the samples if they don't already exist.
        if not os.path.isdir("quickstart"):
                os.system("unzip -n -q GemFire_v6.5_SampleCode.zip")
 
        # Disable verbose output.
        fd = open("quickstart/classes/gemfire.properties", "w")
        fd.write("log-level=warning")
        fd.close()
        doMenu()
 
main()

You need a system with Python, Java and unzip all in your path (no environment variables, please!). Other than that it should work anywhere. Leave me a comment if it you try it and it doesn’t work. Otherwise have fun checking out what GemFire has to offer, don’t forget to check out the continuous query and function execution examples!

Written by Carter Shanklin

December 19th, 2010 at 8:09 am

Posted in Uncategorized

Tagged with , ,