Just in time for the release of JRuby 1.0 and following my experiments with writing JRuby scripts to manage remote applications using JMX (part I & II), I created jmx4r, a simple library which makes it super easy to write such scripts.
For example, to trigger a Garbage Collection on a remote Java application , the whole script is:
By default, jmx4r expects to connect to the remote MBean Server on localhost using the standard JMX Service URL. Otherwise, it is possible to set the host and port of the remote MBean Server:
JMX attributes are available on the JRuby objects but their names are changed to be snake_cased instead of CamelCased. For example, the LoadedClassCount attribute of the java.lang:type=ClassLoading MBean is available as the loaded_class_count attribute on the corresponding JRuby object.
Ditto for the operations:
logging = JMX::MBean.find_by_name "java.util.logging:type=Logging"
logging.logger_names.each do |logger_name|
logging.set_logger_level logger_name, "INFO"
end
The set_logger_level method corresponds to the setLoggerLevel MBean operations.
Compared to previous release, the main new feature is the support for MBean notifications.
Under the hood, the code which displays MBean attribute values has been completely rewritten by Ben Walstrum using extension points. Thanks to his contribution, any MBean attribute display can now be customized (e.g. using charts, hyperlinks, ...)
update: updated Ruby script to use instance_eval and define_method instead of eval based on a poignant explanation of eval-less metaprogramming.
In Part I, I created a JRuby script to manage a Java application using JMX.
In this entry, I'll explain how to remove the dependency on the MBean proxies by taking advantage of Ruby to dynamically add the MBean attributes to a Ruby object representing the MBean.
MBean proxies
Using MBean proxies (using either ManagementFactory.newPlatformMXBeanProxy() or MBeanServerInvocationHandler.newProxyInstance()) is always the simplest way to interact with MBeans. However it also means that the MBean interfaces must be available to the JMX client. This can be sometimes problematic. For example, to use a proxy on your MBean (as opposed to the JVM MBeans that I'm using in my examples for convenience), you must add the jar containing your MBeans in JRuby classpath. If it is not a problem, good news, go for it. However if it makes the setup too complex, let's try to break this dependency.
MBean attributes
To make it simple, I'll focus on the MBean attributes. We will create a Ruby object which makes it simple and elegant to read the value of a MBean attribute without creating a proxy of the MBean (by simple & elegant, I mean something else than calling directly MBeanServerConnection.getAttribute(ObjectName, String)).
We will again use jconsole as our managed java application (see Part I to start jconsole with all System properties required to manage it remotely).
Going directly to the conclusion, the code to display a list of all loggers defined by java.util.logging is
# connect to jconsole MBeanServer
url = JMXServiceURL.new "service:jmx:rmi:///jndi/rmi://localhost:3000/jmxrmi"
connector = JMXConnectorFactory::connect url, HashMap.new
mbsc = connector.mbean_server_connection
logging = MBean.new mbsc, "java.util.logging:type=Logging"
# list all Loggers
logging.LoggerNames.each do | logger_name |
puts logger_name
end
MBean is a Ruby class which creates objects with specific methods to access all the attributes associated to the ObjectName passed in parameter using the MBeanServerConnection to connect to a remote MBeanServer.
What does its code look like?
class MBean
include_class 'javax.management.ObjectName'
def initialize(mbsc, object_name)
@mbsc = mbsc
@object_name = ObjectName.new object_name
info = mbsc.getMBeanInfo @object_name
info.attributes.each do | mbean_attr |
self.class.instance_eval do
define_method mbean_attr.name do
@mbsc.getAttribute @object_name, "#{mbean_attr.name}"
end
end
end
end
end
This class only defines a constructor initialize which accepts a MBeanServerConnection and a String representing an ObjectName which is used to create a @object_name field.
It retrieves the MBeanInfo associated to the @object_name thanks to @mbsc. It then iterates on the MBeanAttributeInfo using info.attributes.
That's in this iterator that the magic happens:
self.class.instance_eval do
define_method mbean_attr.name do
@mbsc.getAttribute @object_name, "#{mbean_attr.name}"
end
end
It calls instance_eval to add a method to the instance class. This method is created using define_method with mbean_attr.name as the method name. It returns the value of the mbean attribute by calling @mbsc.getAttribute for the given mbean_attr.name of the @object_name of the MBean.
What does this mean? Executing logging = MBean.new mbsc, "java.util.logging:type=Logging" will create the logging object and add a method to this object to access the LoggerNames attribute:
def LoggerNames
@mbsc.getAttribute @object_name, "LoggerNames"
end
With few lines of code, it is possible to access MBean attributes in a simple way.
However, to have the same level of functionalities than using MBean proxies, we still need to be able to write attributes and invoke operations. Let's keep that for other entries.
"an open source browser extension that enables web applications to provide offline functionality:
Store and serve application resources locally
Store data locally in a fully-searchable relational database
Run asynchronous Javascript to improve application responsiveness".
At first glance, it seems to be the implementation of what Adam Bosworth envisioned when he was talking about BEA's Alchemy and browser connectivity (search for "When connectivity isn't certain" at the middle of the page).
Gears is already enabled for Google Reader. I wonder how long it will take to have it supported in Gmail...
Another potential use case for Gears is to develop browser-based desktop applications. Their main data store will be provided by Gears database which could be synchronized from time to time to a web store. Office applications (document, spreadsheets, calendar) seem like good candidate for that approach: they'll feel faster and more responsive that way. Synchronizing them will only be done when (auto) saving the document/spreadsheet/event.
Continuing the series about "writing JMX scripts in a dynamic language", after Ruby (part I & II), let's do that in JavaScript.
Aside of the use of a different scripting language, this example differs completely from the Ruby one by its context of execution: it will be integrated into Eclipse and called directly from its user interface (using Eclipse Monkey as the glue).
The example will:
ask graphically the user for a logging level
update all the JVM's loggers with this level
display all the loggers of the JVM
in 50 lines of code.
This example is simple but it implies several interesting steps:
connect to a JMX Server
retrieve a MBean
retrieve value of MBean attributes
invoke operations on the MBean
There are many use cases where you have to perform theses steps in repetition. It's tedious to do that in a JMX console (e.g. jconsole or eclipse-jmx) and most of the time, it is not worth writing a Java application.
These use cases beg to be scripted.
We will again use jconsole as our managed java application (see this previous post to start jconsole with all System properties required to manage it remotely).
Make sure that Eclipse is running on Java 1.5 or above
copy the script just below and save it as logging_change_level.js in the scripts directory of an Eclipse project
click on Scripts > Logging > Change level in Eclipse menu
follow the instruction to install the required DOM
only the JMX Monkey feature is required
restart Eclipse
click on Scripts > Logging > Change level
in the dialog, input the logging level
all the loggers have been updated with the given level
logging__change__level.js
/*
* Menu: Logging > Change level
* DOM: http://eclipse-jmx.googlecode.com/svn/trunk/net.jmesnil.jmx.update/net.jmesnil.jmx.monkey.doms
*/
function main() {
mbsc = jmx.connect("localhost", 3000)
logging = mbsc.getMBean("java.util.logging:type=Logging")
level = ask_level()
set_all(mbsc, logging, level)
text = "All loggers are set to " + level + "\n\n"
text += logger_levels(mbsc, logging)
show(text)
}
function logger_levels(mbsc, logging) {
var out = ""
for each(loggerName in logging.LoggerNames) {
lvl = mbsc.invoke(logging, "getLoggerLevel",
[loggerName],
["java.lang.String"]);
out += loggerName + " is at " + lvl + "\n"
}
return out
}
function set_all(mbsc, logging, level) {
for each(loggerName in logging.LoggerNames) {
mbsc.invoke(logging, "setLoggerLevel",
[loggerName, level],
["java.lang.String", "java.lang.String"])
}
}
function ask_level() {
dialog = new Packages.org.eclipse.jface.dialogs.InputDialog(
window.getShell(),
"Logging Level",
"Which level? (SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST)",
"INFO", null)
result = dialog.open()
if (result == Packages.org.eclipse.jface.window.Window.OK) {
return dialog.getValue()
}
}
function show(text) {
Packages.org.eclipse.jface.dialogs.MessageDialog.openInformation(
window.getShell(),
"Update Logging",
text
)
}
code walkthrough
The main() function of the script is:
function main() {
mbsc = jmx.connect("localhost", 3000)
logging = mbsc.getMBean("java.util.logging:type=Logging")
level = ask_level()
set_all(mbsc, logging, level)
text = "All loggers are set to " + level + "\n\n"
text += logger_levels(mbsc, logging)
show(text)
}
where
mbsc = jmx.connect("localhost", 3000) creates a mbsc object connected to a local JMX server using the standard JMX Service URL.
logging = mbsc.getMBean("java.util.logging:type=Logging") creates a logging object representing the LoggingMXBean used to manage the JVM logging.
level = ask_level() opens an Eclipse input dialog where you can input the logging level and returns it as a String.
set_all(mbsc, logging, level) sets all the loggers at the given level
text += logger_levels(mbsc, logging) returns a String representation of all the loggers and their level
show(text) opens an Eclipse message dialog
The interesting methods are logger_levels() and set_all(). In both methods, we retrieve an attribute of a MBean and invoke an operation.
Let's focus on set_all():
function set_all(mbsc, logging, level) {
for each(loggerName in logging.LoggerNames) {
mbsc.invoke(logging, "setLoggerLevel",
[loggerName, level],
["java.lang.String", "java.lang.String"])
}
}
logging represents a MBean (it is not a "real" MBean, more on that later) and mbsc represents a MBeanServerConnection (but it is not a "real" MBeanServerConnection, more on that later). logging.LoggerNames returns the value of the LoggerNames attribute of the MBean (note that the first letter must be in upper case) which is an array of strings. For each element of this array, we invoke the setLoggerLevel operation using mbsc.invoke(). This method is very similar to the "real" MBeanServerConnection.invoke() method:
something representing a MBean (instead of an ObjectName)
the MBean operation name
the parameters of the operation
the types of the parameters
jmx: an Eclipse Monkey DOM
What do I mean when I write that logging is not the "real" LogginMXBean and that mbsc is not the real MBeanServerConnection?
This 2 types of objects are created by the jmx object in the main() method. This jmx object is in fact an Eclipse Monkey DOM that is contributed by the plug-in listed in the DOM directive at the top of the script:
This plug-in was included in the "JMX Monkey" feature which was installed the first time you ran this script.
The jmx DOM has a single method connect(host, port) which connects to a JMX Server using the standard JMX Service URL. The object returned by this method is a ScriptableMBeanServerConnection. This class encapsulates the "real" MBeanServerConnection (still available using the getObject() method) but only exposes its invoke().
It also exposes a getMBean() method which returns a ScriptableMBean. In turn this class exposes the attributes of the MBean as JavaScript attributes.
To sum up, these are the operations you can perform using the jmx DOM:
connect to a JMX server: mbsc = jmx.connect(host, port)
get a mbean: mbean = mbsc.getMBean(objectName)
get the value of a mbean attribute: val = mbean.AttributeName
get the "real" mbean server connection and use it: objectNames = mbsc.getObject().queryNames(name, query)
invoke an operation on a mbean: mbsc.invoke(mbean, operation, params, param_types)
Conclusion
This script example is simple but quite interesting thanks to its integration with Eclipse.
I believe there is an use for such scripts: repeatable management operations that needs to be tweaked from time to time. It's tedious to do that with a GUI and it's even more tedious to write Java applications to do so.
Last year, during EclipseCon'06, I blogged about an use case for scripting a RCP application using Eclipse Monkey. This is a concrete example: I'm using eclipse-jmx to manage Java applications that I develop. When I realize that I perform the same kind of management task, I write a monkey script which automates it.
Next time, you have to perform the same operation on many MBeans or many operations on the same MBean but you think it is not worth to write a Java application to automate it, ask yourselves if it can not be simply automated by a script such as the one in this post.
[update: fixed Ruby script by using ManagementFactory::newPlatformMXBeanProxy instead of MBeanServerInvocationHandler::newProxyInstance based on Daniel comment]
I needed to write a script to automate the management of a Java application using JMX which I could put in a crontab.
I found some explanation on how to script JMX in jconsole with BeanShell or with Groovy. It's a good approach but:
I want the scripts to be executable in a cron (and not from jconsole GUI)
I'd prefer to write them in Ruby
It turns out that it is straightforward to write such scripts using JRuby and Java 5 or later.
To keep things simple, let say I want to write a script which will be called periodically to run the garbage collector on a long-running Java application.
First let use jconsole as our long-running application. We will start jconsole with all the System properties required to manage its JVM through JMX/RMI connector:
I just need to put the call to this script in my crontab and I'm done.
MBeans in JRuby classpath
Writing such scripts works fine for most cases but there is a caveat: it requires to have the MBean Java class in JRuby classpath to be able to create a proxy using either MBeanServerInvocationHandler::newProxyInstance or ManagementFactory::newPlatformMXBeanProxy (if the MBean is a MXBean as it is the case in this script).
If you have access to the jar containing the MBeans, you just need to put it $JRUBY_HOME/lib/ directory to be able to access it from Ruby scripts.
Still it should possible to map JMX API with Ruby metamodel to get rid of this requirement and develop "JMX-generic" scripts.
Next Step: Generic JMX Scripts
The next step would be to create Ruby objects representing JMX MBeans where their attributes & operations will be added dynamically to the corresponding Ruby instances by examining the MBeanInfo. The previous example would look like:
The big change will be that we will no longer require to have the MBeans in JRuby classpath making it simpler to manage many Java applications (e.g. Tomcat, JBoss, etc.) To be continued...
In eclipse-jmx, a plug-in to manage Java applications through JMX, I have a view which displays registered MBeans using a TreeViewer. Since there can be many MBeans to display (e.g. more than 200 just for Tomcat), it is tedious to navigate in the Tree by expanding many nodes before finding the MBeans I want to manage. To make it more usable, I wanted to add a filter text to the view to show only the MBeans which are interesting to me.
UI-wise, I wanted something like the Filtered Extensions of the PDE Editor:
It shows only the extensions matching the filter text (and also displays the extensions categories).
Since this screenshot was from Eclipse 3.3M1 New & Noteworthy, I thought that this filter was not available for Eclipse 3.2 (or that it was specific to PDE). It turns out that I was wrong, it works on Eclipse 3.2, and it takes only 2 lines to add such behavior to any TreeViewer using a PatternFilter and a FilteredTree (which are from the org.eclipse.ui.workbench plug-in).
Plain Old TreeViewer
The code to create the TreeViewer is straightforward:
public void createPartControl(Composite parent) {
...
viewer = new TreeViewer(parent, SWT.MULTI | SWT.H_SCROLL | SWT.V_SCROLL);
...
}
which, in eclipse-jmx, gives:
PatternFilter & FilteredTree
To filter the tree's nodes, the code to change is minimal:
public void createPartControl(Composite parent) {
...
PatternFilter patternFilter = new PatternFilter();
final FilteredTree filter = new FilteredTree(parent, SWT.MULTI
| SWT.H_SCROLL | SWT.V_SCROLL, patternFilter);
viewer = filter.getViewer();
...
}
With these 2 lines, a filter widget is displayed on top of the tree and the tree's nodes are automatically filtered based on the filter text:
Screenshot has been taken on Eclipse 3.3M5 which uses native search widgets (if one is available) to display the filter control. It works also on Eclipse 3.2 but does not look so good.
By default, PatternFilter will display any nodes of the Tree whose label matches the filter text. By extending it, it is very simple to have more complex filter matching algorithm. It is the case in eclipse-jmx: it displays any node whose descendant's leaves have an ObjectName which matches the filter text.
Kudos to the workbench team for this little UI nugget!
Too much work is done around Eclipse (RCP, Equinox, BIRT, Monkey) that I can't follow all the things which interests me.
However, by browsing the mailing list of the Equinox (Eclipse's OSGi implementation), I discovered a cool new project in the Equinox incubator: Resources Monitoring.
Its mission statement?
To provide a framework for monitoring resources that are contributed by bundles installed on the host machine. The term 'resources' is used to describe something as specific as a single object or something as abstract as an OSGI bundle.
It seems to use JMX to manage the resources and to provide both the server and the client code to manage the resources.
This a long post about designing JMX management on a cluster. To make a long story short: is it a good idea to replicate MBeans on a cluster of Java applications?
I'm working on a Java application which consists of a cluster of nodes, each node running in a separate JVM. Nodes communicates through a common channel through a group communication protocol. Nodes are mostly independent but they are notified when a node joins or leaves the group communication channel.
Each node exposes its managements operations through JMX and contains its own JMX Server which exposes the MBeans of the nodes An administrator can manage each MBeans of the node through JMX using either a RMI or HTTP adaptor:
But the issue is that the cluster does not provide global management. For example, if an administrator wants to manage two MBeans on different nodes to the cluster, he /she has to:
connect to the JMX server of the 1st node
invoke the management operations on a MBean
connect to the JMX server of a 2nd node
invoke the management operations on another MBean
Not very user-friendly...
There are even worse use cases: some management operations should not be performed depending on the state of resources on the other nodes of the cluster.
For example, imagine in the picture above that the MBeans X, Y, Z and T represent the same kind of resource: a Light which can be turned on or off. A mandatory requirements on the cluster is tha there must be always at least one light which is turned on in the cluster at any time.
If the administrator wants to close the light represented by MBean X, he/she has to be sure first that either lights Y, Z or T are still on. Bad news: the management operation is depending on the good will of the admin...
Of course, for our application, we can code a JMX client which does all the grunt work for the admin but putting the management workflow in the JMX client is a bad idea: we open the MBeans to the world and it's likely that some administrators will prefer use a graphical JMX console or a Web one. In that case, they would bypass parts of the management workflow...
In an ideal world, the admin should only ask the light X to switched off and it's up to the cluster to check if it's possible or not.
Of course, we could use the group communication channel to communicate between the nodes of the cluster but I'm not seduced by this proposal. I'd like to keep the management logic as much separate as possible from the business logic which already uses the group channel to exchange data.
Another point: the cluster does not have a central repository to get information or state about the cluster. So it may be interesting to replicate this information on all nodes of the cluster so that whatever node the admin is connected to (through its JMX server), he/she can see the global state of the cluster and perform management operations on any of its nodes.
How do I plan to do that? I was thinking about registering on each node proxies of MBeans whose real implementations are hosted on other nodes:
The dashed rhombus are Mbeans proxies while the plain ones are the real implementations.
In the picture above, when the node #2 joins the cluster, I create proxies of the MBeans registered on existing member of the cluster, node #1 (he received enough information to connect to their JMX servers when joining the group). I register in the JMX server of node #2 the proxies of MBean X and Y with the same ObjectName than the real Mbeans (which are on node #1). Since node #1 is also notified that a new node has joined the cluster, he also creates proxies of MBean Z and T and register them on its own JMX server (still using the same ObjectName than the real ones).
If I go that way, management operations are simplified. To continue on the lights example, the sequence is now:
the admin connect to the JMX server of node #1
it invokes the off() operation on Mbean X
the implementation of MBean X will query the JMX Server of node #1 about all the Light Mbeans and checks there is still one which is on (excluding itself)
when it checks on MBean Z, the proxy will connect to the JMX server of node #2 and invokes the method on the real implementation of MBean Z
(1) from the client, connect to the JMX Server of node #1
(2) get the MBean X and invoke one of its method
(3) get the MBean Z and invoke one of its method
(3') the proxy of MBean Z will connect to the JMX Server of node #2
(3'') the real implementation of MBean Z method will be invoked
The cool thing about this design is that the admin does not even need to be connected to the JMX Server of node #1 to manage MBean X:
(1) from the client, connect to the JMX Server of node #2
(2) get the MBean X and invoke one of its method
(2') the proxy of MBean X will connect to the JMX Server of node #1
(2'') the real implementation of MBean X method will be invoked
(3) get the MBean Z and invoke one of its method
The code is the same regardless of the node the administrator is connected (since the proxies and the real implementations share the same ObjectName on all the JMX Servers of the nodes of the cluster). And the whole design is quite dynamic since each JMX Server can also be notified when MBeans are registered or unregistered on other JMX Servers of the cluster
However I already see some issues with that approach:
what about security credentials when forwarding an invokation from one JMX Server to another?
what about the scalablity if the number of nodes or MBeans increases?
what about event notifications? I'm interested to be notified of events emitted by the real MBeans, not the proxies. How do I make the difference when registering NotificationListener based only on the ObjectNames wich are identical?
and the first question I asked you at the beginning of this post: is it a good idea to replicate MBeans on a cluster of Java applications?
Are theses issues showstoppers either architecturally or technically?
I did some searches on the web about this kind of approach with regards to JMX but I did not find anything conclusive so far so I plan to code a prototype to handle this stuff and see where it leads me. [Update: Coherence seems to provide this feature.]
Wow! Still with me after this long post? In that case, I'd be happy to talk about this design with you. Please do not hesitate to post a comment or send me a mail.
