Protip: reading to the end will yield a sneak peek of a new, upcoming Splunk app!
The Background
Almost 10 months ago, Splunk chose Okta as its federated identity management and single sign-on (SSO) vendor. There were several benefits from this project including multifactor authentication (MFA) for our business applications and VPN, user experience enhancements by not requiring Splunkers to remember multiple passwords, and instant deprovisioning once an Active Directory account was terminated.
As part of our ongoing efforts to make Splunk’s instance of Splunk (affectionately dubbed “Splunk(x)”) more valuable to the business, we made the decision to provision multiple, purpose-built search heads. We have a search head that serves as a primary point-of-entry, a search head for our Enterprise Security app per best practice, a jobs scheduler, and several others.
To provide a better user experience and better secure Splunk(x) through MFA, we decided to onboard these Splunk search heads to Okta to provide a seamless SSO experience when transitioning between search heads. To achieve this on the Splunk side, we had to implement Apache as a reverse proxy leveraging mod_auth_mellon and lasso to consume and validate the SAML 2.0 assertion generated by Okta.
![]()
Indira wrote a blog post outlining a high-level overview of what must transpire to allow Splunkweb to consume SAML assertions.
Building upon that overview, we’ve documented the process we took to onboard Splunk running on RHEL 6 to Okta using SAML v2. With modification, these steps could apply to any SAML Identity Provider (IdP).
The Procedure
- Before setting anything up, ensure you have a proper LDAP strategy configured and roles mapped if you will be using the same LDAP as Okta authentication. Splunk must know about valid users to let them in with the SSO header that mod_mellon will set. You should do this before continuing as it will be difficult to configure LDAP once you enable SSO. You can find documentation on configuring LDAP at the following URL: http://docs.splunk.com/Documentation/Splunk/latest/Security/SetUpUserAuthenticationWithLDAP
- Ensure yum repositories are up to date. You’ll need several packages and their dependencies.
- Install the following packages from yum, in order. Use yum install and the package will be updated even if its already installed.
- httpd
- xmlsec1
- xmlsec1-openssl
- xmlsec1-openssl-devel
- mod_ssl
- openssl
Download and install the RPMs for lasso and mellon from http://dev.entrouvert.org/redhat/6/RPMS/x86_64/. Install them in order; mellon requires lasso.
Ensure you open ports 80 and 443 in iptables. During testing, leave 8000 open as well so that you can debug the SSO settings if anything goes wrong. Best practice is to block port 8000 once the configuration works. To do this, look for the following line in /etc/sysconfig/iptables:
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
And add these lines directly after it:
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8000 -j ACCEPT
Restart the firewall: service iptables restart
Create mellon_create_metadata.sh from the attached file. Ensure you set the execute bit (chmod u+x mellon_create_metadata.sh)
Create /etc/httpd/mellon (or a similar location to store certificates and metadata files)
Edit the certificate files paths in /etc/httpd/conf.d/ssl.conf to point to your certificates.
Add the following lines to $SPLUNK_HOME/etc/system/local/web.conf under the [settings] stanza. Replace [YOUR_SPLUNKWEB_IP] with the IP address to which users will connect to access Splunk.
trustedIP=127.0.0.1,[YOUR_SPLUNKWEB_IP]
remoteUser=SplunkWebUser
SSOMode=permissive
To require SSO, ensure you set SSOMode=strict. Permissive will attempt to log in an SSO user and fall back to Splunk auth if not found.
Add the following line to the [general] stanza of $SPLUNK_HOME/etc/system/local/server.conf. Note that you do not need to add the IP address to which visitors will connect; just the loopback address.
trustedIP=127.0.0.1
Restart Splunk and verify that splunkweb is bound to port 8000
Create /etc/httpd/conf.d/auth_mellon.conf from the attached file.
Use the following command to create the metadata and certificates for mod_mellon. Replace [YOUR_SPLUNKWEB_URL] with the URL at which users will access Splunk, e.g. splunk.yourcompany.com.
./mellon_create_metadata.sh urn:splunkweb:[YOUR_SPLUNKWEB_URL] https://[YOUR_SPLUNKWEB_URL]/secret/endpoint
The preceding step will generate three files. Move these to /etc/httpd/mellon, and update your auth_mellon.conf MellonSP* variables to point to these three files.
Update the ProxyPass settings in auth_mellon.conf to point to [YOUR_SPLUNKWEB_URL]:8000
Warning: IF YOU ARE RUNNING MULTIPLE SPLUNKWEBS it is vitally important that all have a distinct MellonVariable value (e.g. “splunkweb1-cookie”, “splunkweb2-cookie”). Failure to assign distinct values may result in infinite login redirect loops if multiple tabs are open.
Create a Template SAML 2.0 app on Okta SANDBOX![]()
Post Back URL, Recipient, and Destination should all have the same value:
https://[YOUR_SPLUNKWEB_URL]/secret/endpoint/postResponse
Audience restriction should have the following value:
urn:splunkweb:[YOUR_SPLUNKWEB_URL]
Default Relay State should be set to the root of the domain (/)
Name ID Format should be Transient
Do not assign the app to any users yet. After saving the changes, change default username format from Okta username (default) to AD sAMAccountName.![]()
Also on the Sign On tab, you will see a link to SAML 2.0 setup instructions. At the bottom of the page to which that links, you will see an XML document in a textarea input (IDP metadata). Get IDP metadata and copy it to add to /etc/httpd/mellon/okta.xml
Verify that ntpd is running and that the date on the server is correct. An incorrect date will not only cause inaccurate search results but may invalidate SAML assertions based upon the NotBefore or NotAfter parameter. If the date is inaccurate:
- Stop ntpd (if it is running): service ntpd stop
- Issue the command (as root or sudo): ntpdate -s time.nist.gov
- Start ntpd: service ntpd start
- Ensure ntpd is set to start with the system: chkconfig –add ntpd
Start httpd: service httpd start
Assign to yourself in Okta SANDBOX
If you have any issues logging in, check /var/log/httpd/error_log and /var/log/httpd/ssl_error_log. These two files will generally contain the diagnostic information needed to troubleshoot. Resolve any issues and proceed once working in Okta SANDBOX.
Clone chiclet in production, use the same values we used in Post Back URL. Recipient, Destination, Audience Restriction, Name ID Format, and Default Relay State. Again, DO NOT assign to a user; save the chiclet.
Update the username format to AD sAMAccountName under the Sign On tab.
Grab the IDP metadata file (see step 25) and replace /etc/httpd/mellon/okta.xml with the production XML.
Restart httpd: service httpd restart
Assign chiclet to groups in Okta and test the SSO. If everything is working, proceed. Otherwise, troubleshoot using the log files in step 29.
As a best practice in user experience, add the attached redir.conf to /etc/httpd/conf.d/redir.conf. This will redirect all non-SSL connections to SSL (http to https).
Restart httpd: service httpd restart
Ensure httpd is set to run at startup by issuing the following commands:
chkconfig --add httpd
chkconfig --level 2345 httpd on
Congrats! Your Splunk is now in Okta!
Files
Please ensure you change the file extension on the following files as noted in the step-by-step!
mellon_create_metadata.sh: http://blogs.splunk.com/wp-content/uploads/2013/10/mellon_create_metadata.txt
auth_mellon.conf: http://blogs.splunk.com/wp-content/uploads/2013/10/auth_mellon.txt
redir.conf: http://blogs.splunk.com/wp-content/uploads/2013/10/redir.txt
Sneak Preview!
As part of the work we’ve been doing with Splunk and Okta, the Splunk(x) team has been working on a Splunk App for Okta leveraging the power of the Splunk SPL and reporting on an Okta data set. Here’s a preview screenshot of the app — coming soon to Splunk Apps!
![]()
