February 23, 2015
Summary: The next course will be about clojure.test. Sign up to be notified when it is published.
Last week I mentioned my new course format, and I also teased at more announcements. Well, it's a course on clojure.test, the Clojure testing library that comes built in with Clojure.
I chose clojure.test as my next topic for a number of reasons. The main reason is that I think it's a good idea to begin with the fundamentals. I have a long list of topics, and testing was as good as any. It's also a video that opens doors to other topics, including testing using Midje, test.check, and simulation testing.
The course is called LispCast Intro to clojure.test. clojure.test is not a big library, but it's important to know. This course will teach you how to use the library and conventions around testing that might not be so obvious. For instance, how to name your testing namespace.
The course is built in the new format, and it's shorter than my previous courses. It's common for someone to tell me how much they like my courses. My first question is always "Did you finish them?". And it disturbs me how many people say "no". They like the course but they don't finish. So I'm doing smaller courses that are easier to fit into your day and finish. I expect to make more, smaller courses from now on. And Intro to clojure.test will be the first.
And now that that announcement is done, I must tease you with just one more announcement coming later this week. If you don't want to miss it, sign up for the mailing list below. You'll be the first one to know when clojure.test is coming out and what that cliffhanger announcement is.
You might also like
March 30, 2015
Summary: Where to put your tests is a common question. You could put them anywhere, but you want to pick a place that makes it easy to find, easy to exclude from production, and work well with your tools. My recommendation is to follow what most projects do, which takes care of all of these requirements.
You want to write some tests in Clojure. Maybe they're unit tests. Maybe they're integration tests. The first question you must answer is where do you put your tests?
And you don't want them just anywhere. You actually have some important requirements dealing with where they are:
- They should not be compiled/run on production.
- They need to be easily findable.
- They should be runnable from the command line.
- They should be runnable from the REPL.
Here's my recommendation, which is the de facto standard of organizing your tests. It works with Leiningen, CIDER, and vim-fireplace.
First, you make a new namespace structure in a test/ directory. It should mirror the src/ directory.
If you have:
src/
lispcast/
core.clj
init.clj
util.clj
Then your test directory should look like:
test/
lispcast/
core_test.clj
init_test.clj
util_test.clj
But also notice the second point: that the structure is the same, but the names are slightly different, but it a systematic way. It's really easy: you just add -test to the namespace name, which becomes _test in the file name. Then, you put all the tests that test lispcast.core into lispcast.core-test. Now they're easy to find!
If you need to write a test that crosses two different namespaces (like an integration test might), then you can just make a new test namespace that doesn't correspond to one or the other.
Leiningen will load the test/ directory selectively, depending on if you're deploying to production (it won't load test/) or running the tests (it will load test/).
So, it's that easy. You are free, of course, to put the tests wherever you like. But this is my recommendation!
If you're getting into testing in Clojure, you should check out LispCast Intro to clojure.test. It's an interactive course. It has animations, screencasts, exercises, code samples, and text.
You might also like
March 28, 2015
Summary: LispCast Intro to clojure.test will launch this week.
LispCast Intro to clojure.test is ready for launch! It's been incubating in the PurelyFunctional.tv Early Access Program (PEAP) for a month. People who opted in to the PEAP got an early version of the course, have been asking questions on the forum, learning a ton, and having a direct impact on the quality of the course. I'm so glad I did it this way. It let me test out the new course format and work out the kinks with the help of my customers.
And now it is clear the new course format has performed really well. It's very interactive. It contains exercises, animations, screencasts, code samples, and narration. All directed at getting clojure.test installed in your brain, best practices, conventions, and all.
It's suitable for beginners to Clojure and for anyone who wants to learn how to properly use clojure.test. clojure.test is the main testing library that comes with Clojure and works well with Leiningen, CIDER, vim fireplace, midje, test.check, and more!
The course will launch this week. Sign up to learn when it comes out.
You might also like
September 26, 2014
Background
Ashton Kemerling's talk at the conj is about Generative Testing (also known as Property-based Testing) as applied to integration tests. Generative Testing is a methodology for generating many tests randomly to find failing cases (as opposed to writing each test by hand). It often finds difficult bugs and can present a very small, reproducible failing case.
The best talk I have seen about Generative Testing is John Hughes' talk at Clojure/West. John Hughes wrote Quickcheck for Haskell and Erlang. Reid Draper has recently ported it to Clojure in the form of test.check. He also gave a talk at Clojure/West about test.check.
Why it matters
Generative testing is great for unit testing. It has already found bugs in Clojure itself. What's not obvious is how to apply the technique to integration tests to test the interaction of different systems. This talk appears to be an experience report from a company that has actually used Generative Integration Tests.
Twitter - Github - Blog
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
You might also like
October 20, 2014
Introduction
I asked Ashton Kemerling some questions about his talk at Clojure/conj about using generative testing in Clojure. Read the background to his talk.
Interview
LispCast: How did you get into Clojure?
Ashton Kemerling: The path I took to Clojure is fairly roundabout. During my College years I was very interested in function programming, although ML & Haskell were my favorites back then. I think a large amount of my fascination with them stemmed from the fact that I had more experience than my classmates, my high school had a very rich set of programming courses, and I needed something interesting to play with while taking introduction to OOP courses.
Fast forward a few years and I ended up working at a job using Common Lisp professionally in a legacy web application. The application had a lot of problems due to the condition of CL libraries at the time and we were looking for something to switch to. Clojure was an obvious choice to look at. It's still a lisp, so a lot of our habits would still work, but it fixes the deployment & library issue that CL has while adding a much richer set of data structures. Unfortunately our application relied extensively on mutation (CL doesn't really encourage or discourage any coding style) and OOP, so a conversion was deemed too costly to attempt.
Over the course of the following years I continued to tinker with Clojure on the side. I was convinced it was a great way to do things, but I could never convince anyone to consider using it in their code base. Most places were either too invested emotionally in OOP, or had well tested and working codebases that made a rewrite unwise. So I mostly just hacked on side projects using Clojure over the intervening years without using it professionally. But in the last year or so I realized that Clojure makes a great secondary language for odd-tasks in non-Clojure codebases, which brings me to the current state of using Clojure to test a very large Ruby on Rails and Javascript application.
LC: Can you give a basic outline of how your Clojure program tested the web front end?
AK: We've basically tested in two ways. The first was to simply compile our JS file along with Clojurescript tests and run them in a PhantomJS process similarly to how you'd run Clojurescript or Javascript tests normally. We're still working on this approach, but it's slower to work on that I would have hoped.
The second way was to leverage the fact that Clojure's on the JVM. This involves mixing JDBC, HTTP libraries, and Selenium in novel and exciting ways. This has been the most fruitful way of testing, and what I'll be focusing the most on during my talk. We've used the code I've written as both a tool to hunt down API mistakes, and as a means of narrowing down the reproduction steps on complicated bugs reported by the user.
LC: Can you explain a little more about that? What do you mean by "narrowing down reproduction steps"?
AK: Sure. I'm testing Pivotal Tracker, a very large (~24kloc) Javascript "single page app". With large apps like this there are ways for the application to subtly fall out of sync with the server or process data in a degenerate way.
In particular a lot of times you'll get a report from a user that says something along the lines of "I reloaded and it crashed". You can dig up the logs from that session, but you'll get the last 20 or 30 steps they did before the crash, which doesn't really help you out a whole lot. So we occasionally turn the generative tests towards a type of action (in this case, do things and then refresh) in the hopes that it will help us narrow down to the minimal reproduction steps required to trigger a problem.
In the case that happened last week, we were able to find 2 distinct actions that would provoke the issue reliably after about 20 minutes of test modification and running. Obviously this took hours off of the process required to find the actual root of the bug, since 2 is orders of magnitude easier to work with than 7 or 10 when debugging.
LC: I see. So how is test.check able to reduce the reproduction steps?
AK: Test.Check provides "shrinking". Just as the generators used to create randomized data have the ability to produce more complicated data, they also have the ability to produce less complicated data. Test.Check records all of the failures it finds and attempts to simplify them and find the smallest failing case it can.
LC: A lot of the Clojure/conj participants will be new to Clojure. What resources would you recommend them to make the most of your talk?
AK: They can't go wrong with the test.check README or any of the blog posts mentioned therein. I recommend reading the source directly because it's heavily commented, in particular I recommend the generator source, because the generators represent 90%+ of the test.check API a user will interact with.
Beyond that I have blog posts on my work blog and on my personal blog. Also Reid Draper and I were both on the Cognicast, both of which are based around this subject.
LC: Are there any resources on Selenium and other methods for running frontend tests?
AK: The Clj-Webdriver docs are all I can recommend.
LC: Where can people reach you?
AK: Twitter: @ashton
My blog: ashtonkemerling.com
LC: If Clojure were stranded on a desert island, what one book would it bring?
AK: A boat building book, clearly.
LC: Awesome! Thanks for the interview, it was a pleasure.
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
You might also like
April 05, 2015
This summary was graciously written by Nola Stowe. She's a programmer, the co-founder of DevChix, and a prolific teacher. She recently ran ClojureBridge Austin. Please shout out to her and say thanks!
Ryan Neufeld's talk at Clojure/West is about Simulation Testing using Simulant.
Background
Simulation Testing is a systems based testing methodology where a series of actions are randomized but repeatable and then the outcomes are recorded and observed. The actions can simulate realistic load on the system. The library for that in Clojure is Simulant.
Hear more information on episode 59 of the Cognicast where Michael Nygard talks about using Simulant. When researching this topic I found what looks to be a simple example testing an API with simulant.
Homepage - Github - Twitter
Ryan is the co-author of: Clojure Cookbook
You might also like
March 26, 2015
Summary: Unit testing in Clojure is straightforward. Here are a few testing ideas as they apply to Clojure.
Most of the Unit Testing literature discusses how to unit test Object Oriented code. However, Unit Testing is very useful for functional code, including Clojure. I'd like to document a few unit testing ideas as they relate to Clojure.
The Unit you're testing is the function
Unit testing is about testing the smallest unit of code. In Object Oriented languages, the unit is the class. In functional languages, the unit is typically the function. This is true in Clojure. If you're testing individual functions, you're unit testing Clojure.
Example-based tests
The easiest kind of tests to do is example-based, which means you test that for a given argument, you get a known return value. Let's look at a simple example:
(deftest addition-tests
(is (= 5 (+ 3 2)))
(is (= 10 (+ 5 5))))
You're testing that + works on two different inputs! Notice: 2 lines and 2 inputs, it looks like we'll get linear growth in tests as our coverage increases.
Round-trip testing
Ok, it's not exactly unit testing, if you are strictly going by the definition of "unit", because you're actually testing two functions. But who's really being so strict? A really useful kind of test is the round-trip test.
In Clojure, pr-str prints a value readably, meaning if the value can be read back in, this will make a string that could be read in using clojure.edn/read-string. You can do a round-trip from value to string back to value, and the two values should be equal. You're testing the property that these two functions are inverses of each other.
Example:
(deftest pr-str-read-string-round-trip
(is (= [1 2 3] (read-string (pr-str [1 2 3]))))
(is (= {:a :b :c :d} (read-string (pr-str {:a :b :c :d})))))
Again, we're getting linear test growth.
Here's another example where I test that addition is the opposite of subtraction:
(deftest +---round-trip
(is (= 5 (-> 5 (+ 10) (- 10))))
(is (= 10 (-> 10 (+ 100) (- 100)))))
What examples to test
If you're writing example-based tests one-by-one, and you're getting linear benefit for your examples, you've really got to maximize what you test, because linear growth is actually quite bad. In that case, what do you test? The best thing to test are the corner cases. Corner cases are mostly domain-dependent, but there are some domain-independent ones.
Empty collections
It's good to test what happens when you call a function on an empty collection. It could be that you didn't handle that, or didn't handle it correctly. The biggest gotcha is stuff like dividing by the size of the collection. If it's empty, the size is zero, and that's undefined.
Empty strings
The bane of the web programmer's existence, empty strings are usually not valid input, but of course that doesn't stop someone from passing one in. Are you testing that it's valid?
Zero
Zero is actually a typical corner case. Try it out.
One
One is also a typical corner case.
Normal cases
You should have at least one normal case to test the expected behavior. A normal case is a list with 5 elements, or a small integer (7, 12, 34).
Bugs
Now, here's the thing that makes having a test system set up totally worth it: having a place to put test for known and fixed bugs. If someone reports a bug, it's really nice to reproduce it in code in a failing test before you fix it. If the bug happened once, it could happen again, so make it part of your anti-regression suite.
Multiple assertions on the return value
One last thing that happens in Clojure is you want to assert a few things about the same return value. Instead of calling the function several times, why not save the value and assert a few things about it?
(deftest map-test
(let [return (map - [1 2 3 4 5 6 7])]
(is (= 7 (count return)))
(is (every? neg? return))))
One step further
Ok, I've mentioned a few times that example-based testing does not scale. Code coverage grows linearly as the number of examples grows. How do you get better than linear? One way is to use property-based testing (also known as generative testing). Instead of the programmer giving examples, the program generates examples! Here's a preview:
(defspec edn-roundtrips 50
(prop/for-all [a gen/any]
(= a (-> a prn-str edn/read-string))))
This tests that any value (gen/any) can be printed to a string and read back in, and you get an equivalent value. Three lines. You can run this with as many randomly generated values as you'd like. Thousands. Millions. With three lines. That's leverage.
Conclusion
Ok, those are just a few ideas that could get you started with example-based unit testing in Clojure. If you'd like to start automated testing in Clojure, I suggest you check out LispCast Intro to clojure.test. clojure.test is the built-in, standard Clojure testing library that most systems use (or are compatible with). The LispCast course is an interactive course with animation, examples, screencasts, text, code samples, and more!
Now, if you'd like to up your game at testing, and want to get more than linear bang for your buck, you've got to get into generative testing.
You might also like
