Trigger an Octoblu IoT Flow from Splunk

Octoblu does some incredible stuff with physical things driven by software. Steve Greenberg and I did a demo during Geek Speak Tonight at Citrix Synergy triggering physical devices via a Splunk search. This article goes into some of the Splunk details.

An alternate title to this article might be “How Steve Greenberg and I Pulled off the Robo-Kitty Monitor Alerts at Citrix Synergy Geek Speak Tonight”.

In case you missed it, here it is below:

 

The Demo Scenario

Anyway, Octoblu does some incredible stuff with physical things driven by software. In the demo that Steve and I did, Steve set up an actual desk with various IoT devices on it.  We will call this the “IoT Workspace”.  The IoT Workspace had a digital picture frame, lights that can change color, a mini file cabinet (that held business cards), a maneki-neko (a.k.a. lucky cat) with several hacked features, a smoke machine, and more.  We started the IoT Workspace using proximity (look for another article about how we did that soon). Then, I had a Splunk instance monitoring a Citrix stack including network, XenApp, hypervisors (XenServer, Hyper-V, and VMware), physical hardware (in this case Cisco UCS), shared storage, NetScaler, etc. At the beginning of the demonstration, the environment was all-good so everything glowed green, the Robo-Kitty was happy, and the picture frame on the desk showed an array of our favorite pictures (see below):

IoT Workspace
IoT Workspace
Splunk Octoblu Green
Splunk Dashboard (All Good)

 

Then, the ICA Round Trip Time started to go up. Nothing terrible, but we used that as an indicator that our users might start seeing some lag in their sessions. This is where Splunk fired the first Octoblu trigger to go to a “yellow” state. The picture frame showed a worried Minion, lights turned yellow, and Robo-Kitty’s eyes turned yellow and started to swivel.

Next, Splunk showed problems with the XenApp servers and hypervisors in addition to the ICA Round Trip Time. So, Splunk triggered a “red” state. The Minion in the picture frame looked more worried, lights turned red, and Robo-Kitty’s eyes turned red and around faster.

Finally, the entire stack went to pot. The write latency on the storage array went through the roof, the hypervisors were not happy, the XenApp server resources were scarce, the ICA Round Trip Time was off the chart. Splunk triggered the “defcon red” state. Robo-Kitty shot lasers out of its chest, the storage cabinet on the IoT Workspace started to rattle, smoke was coming out of the desk. The culprit ended up being a write controller issue on the shared storage.  Once everything was fixed, Splunk triggered the “green” state again.

 

How it Works

Steve did all the physical work building the IoT Workspace by hooking up Raspberry Pi, Gateblu, servos, lights, etc. Check out his article for more info -> http://www.thinclient.net/blog/?p=473. Steve also built the Octoblu flows to make all that stuff work.

I hooked up the Octoblu triggers in Splunk to kick off all these connected devices. A trigger generally initiates Octoblu flows. These triggers have HTTP POST URLs that can be used to remotely initiate the flows (see screen shot). This is how I had Splunk act upon the data seen in the Citrix stack.

Octoblu Trigger

 

 

I ran a Splunk real-time search and triggered a Python script that initiated the HTTP POST with data from the Splunk search to Octoblu when certain conditions happened. For example, if the ICA Round Trip Time exceeds 30ms and is less than 60ms, trigger a yellow alert condition. Here is the Splunk search:

sourcetype=ICA:RTT ICARTT > 30 ICARTT < 60 | eval url="<HTTP POST URL for the Octoblu trigger>" | eval alert_level="Yellow"

If you are interested, I have the entire Splunk/Octoblu example I used uploaded to GitHub. You can also download and use Splunk for free.  There is a data generator built in there as well that will let you trigger different conditions like I did in the demo.

The magic happens in the saved search and a python scripted named octoblu_trigger.py.  The saved search contains the HTTP POST url and the condition.  The python script takes those parameters from the search and sends it over to Octoblu.  If you want to play around with this on your own system, be sure to edit the saved search by opening the Octoblu app in Splunk and clicking Settings -> Searches, reports, and alerts:

Splunk Octoblu Search SettingsClick on the example alert and change the URL to your Octoblu trigger URL.

Conclusion

Anyway, there you go. It may look kind of complicated at first, but really it is quite easy to trigger any Octoblu workflow given a variety of trigger situations.

 

 

From WIRM to Splunk – Translating the Past to the Present

I get to do a lot of cool things at Splunk.  One of the things I have been wanting to do is incorporate the visualizations I built a long time ago for Web Interface for Resource Manger in Splunk applications.  All of those past visualizations were built using Microsoft ASP.NET and Flash.  So, I have had to use alternate methods to accomplish what I want.

 

Calendar Visualization

One of the first things I tackled was the calendar visualization that shows how many users log in each day in a month.  Here is the old calendar from WIRM:

WIRM Calendar

Here is what the calendar looks like in Splunk:

Splunk Calendar

If you want to see how to use this calendar visualization in your own Splunk environment, check out my Splunk blog post.

The cool thing about having this available in Splunk is that it is reusable for various types of data including security, XenDesktop/XenApp, Microsoft Windows, Unix, etc.