Serving Up a robots.txt File with Spring MVC

Since we now have Google Analytics in our NixMash Spring app it’s time to add a robots.txt file! We’ll see how we do that with Spring MVC…and Velocity.

“Velocity? Are you serious?”

I heard that.

While it might sound like overkill you’ll see how well it works for us. Besides, since we have it with our Spring Mail Module we might as well use it for other things.

First our beautiful robots.txt file.

As a sidebar, if we wanted to serve up a bunch of text files in our Java Web app, we wouldn’t use Spring at all. The smart play would be to use Apache Proxy Mod and store the files in a physical location outside of the app, or at least outside of its Java structure. I mentioned Proxy Mod before. Here’s a post on it that shows you what I mean. If we would do a Proxy Pass on a /files directory the result would look like this.

We want to serve up robots.txt in the root and use some good Spring JuJu along the way, so let’s start in our Controller method. We set the response to text/plain – UTF-8 encoding then call a getRobotsTxt() method in a Spring templateService Component and we’re done. If you didn’t want to go with a Component and/or Velocity, you could replace the last line with any text source.

Swingin’ with Velocity

Here’s the simple call to our velocityEngine Component. This differs from our usage with mail only in that we’re not passing a Model to be merged. The robots.vm Velocity Template which we don’t have to display is a simple text file, our robots.txt file.

Security Settings

Since we’re using Spring Security we need to make sure that bots can access the file, so we add it to our Ignored Resources List.

Don’t Forget the @Tests!

Okay, okay! We’ll write a test to make sure our robots.txt file is served correctly!

Source Code Notes for this Post

All source code discussed in this post can be found in my NixMash Spring GitHub repo and viewed online here.