April 08, 2015
Tyler Tallman's talk at Clojure/West is about how to model complex problems in Clojure.
Background
Medical care is incredibly complex, and is only getting worse. Most medical record software is like a basic CRUD app. It stores the information the doctor puts in and displays it when asked. But as the amount of information increases, something more needs to happen. How can the computer system support the doctor making ever more complex decisions? How can it support the host of medical professionals that provide care? And what role can Clojure play in that process?
I don't have any background information for this one. But I know Tyler personally and this is a fascinating story of rules engines, complex data formats, visualizations, and composable abstractions.
GitHub - Twitter
This post is one of a series called Pre-West Prep, which is also published by email. It's all about getting ready for the upcoming Clojure/West, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-West Prep is about. I want to enhance everyone's experience at the conference by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations.
Clojure/West is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/West or Cognitect. It is simply me (and helpers) curating and organizing public information about the conference.
You might also like
July 22, 2014
Summary: Ring is great because it closely models the HTTP message format using native Clojure data structures. It strictly defines a message format that any software can use and rely on. With Ring 1.3, the specification has gotten even closer to the HTTP spec.
A couple of months ago, Ring 1.3 was released without much fanfare. It included a few improvements and updates, but in general, not much had changed.
One change, though, is very significant: the specification is shorter. It's simpler. Three keys were deprecated in the Ring request map (:content-type, :content-length, and :character-encoding). These keys were unnecessary because their values were in the headers, which are also in the Ring request. Equivalent utility functions have been added for pulling the data out of the headers.
Why is this important? While many libraries get more complex and overburdened, it is refreshing to see a library going in the correct direction of shedding complexity. It does not significantly impact application development. Nor does it reduce the already low barrier to entry. Still, I welcome this kind of change.
Ring is the central specification that ties most of the Clojure web ecosystem together. The spec should be minimal. And a mark of good software is that it models the problem very closely without unnecessary abstraction. Ring merely defines a common format (using Clojure data structures) that mirrors the text-based HTTP message format. That's why Ring has worked so well thus far and why it is appreciated in Clojure.
Because I was so happy about the change, I decided to update my Ring Spec to Hang on the Wall PDF. The newly deprecated keys are gone. It used to be two pages long. The Ring Request took up an entire page, and the Response took up about half of one. But now, with three keys removed and a little tweaking of the font sizes, everything fits on one page.
One page in big, readable fonts, with just the information you need for quick reference. I like it. I'm printing one out right now to tack on the wall. You can get a free copy for yourself by getting on the PurelyFunctional.tv mailing list here.
You might also like
March 28, 2014
Summary: The Ring SPEC is the core of the Clojure web ecosystem. The standard is small and a reference is handy.
If you program the web in Clojure, you probably use Ring. Even if you don't, your server is likely Ring compatible.
Ring has a small SPEC. It's centered around defining the keys one can expect in the request and response maps. And the exact names for keywords are easy to forget.
I don't want to forget. I use Ring often enough that I want a quick reference. A while ago, I printed out a quick summary of the keys for the request and response maps and hung it on the wall behind my monitor. I refer to it frequently.
If you program the web in Clojure, you might appreciate this printout. If you're learning, it could be an invaluable reference. Please download the PDF.
And if you like it, you might also like my Web Development in Clojure video series.
You might also like
November 10, 2014
Summary: Token Bucket is a simple algorithm for rate limiting a resource. It's easy to understand because you can reason about it in terms of real-world objects. core.async makes this algorithm very clear and easy.
You know you've got a good abstraction when you find lots of algorithms that are easy and clear to write using them. One algorithm that I really like because it's practical and simple is called the Token Bucket. It's used for rate limiting a resource. And it's very easy to write in core.async.
Let me digress a little. Here in the US, we have a complicated voting system. Not only do your presidential votes count only indirectly, but the counting is done with machines that vary from state to state. Some of those systems are so complicated. It's a little scary because it's hard to know how those machines work.
I mean, relatively hard. It's obviously possible to know, if you study enough. But compare it to papers in a box. I know there's all sorts of room for corruption in the papers in a box system. But it's understandable by children. Well, as long as they can count.
The token bucket is similarly simple. There's a bucket. You put tokens in it at a certain rate (for instance, once per hour). If you have a token, you can take an action (ride the subway, send a packet, read a file), then you lose the token (it's good for one ride). If you don't have a token, you can wait. Everything slows down to the rate of tokens falling into the bucket.
It's so simple. It so easily corresponds to a real-world situation you can imagine. I love that type of algorithm.
An implementation
Let's build Token Bucket in core.async.
First, we need a bucket. For that, we'll use a core.async channel with a buffer. Let's just start with size 10.
(def bucket (chan 10))
Bucket is done. Now, we need something to add tokens to the bucket at a given rate.
(go
(while true
(>! bucket :token)
(<! (timeout 1000))))
That will add one token to the bucket every second.
We can rate limit an existing channel by forcing it to take a token before values get through.
(defn limit-rate [c]
(let [out (chan)]
(go
(loop []
(let [v (<! c)]
(if (nil? v) ;; c is closed
(close! out)
(do
(<! bucket) ;; wait for a token
(>! out v)
(recur))))))
out))
Corner cases
Ok, it's not that simple. There are two corner cases.
- What happens when nobody takes a token out of the bucket? Do you keep putting coins in?
The answer is yes. In the next hour, there are two tokens, so two can come through. But then . . .
- What do you do when the bucket gets really full and a whole bunch of people take tokens out at the same time?
Well, you let them all through. One token, one ride. There's no other coordination. And that means it's really important to choose the size of your bucket.
The number of tokens your bucket can hold is called the burstiness. It's because when the bucket is full, you could get a rampage of people trying to get on the subway. How many people should be allowed through at that point? The burstiness is the maximum that should come through at a time.
We have our two parameters: the rate and the burstiness. Let's incorporate all of that.
(defn limit-rate [c r b]
(let [bucket (chan b) ;; burstiness
out (chan)]
(go
(while true
(>! bucket :token)
(<! (timeout (int (/ 1000 r)))))) ;; rate
(go
(loop []
(let [v (<! c)]
(if (nil? v) ;; channel is closed
(close! out)
(do
(<! bucket) ;; wait for a token
(>! out v)
(recur))))))
out))
The burstiness is taken care of in the size of the buffer. The buffer will fill up if no one takes a token. Since we're using blocking buffers, putting tokens into a full bucket will block until something takes a token--exactly as the algorithm describes.
Well, that's it. It's easy. And now we have a way to limit the rate of a channel. We can use it to limit the rates of other things, too, like a function call or access to a database. I use it to rate limit an API.
core.async makes this algorithm nice and easy to use. There's a library that does this for you in a very convenient package. It's called Throttler. Bruno Vecchi has done the work of making this work well as a library. If you'd like to learn core.async, I recommend my LispCast Clojure core.async videos. It's a gentle and fun introduction to a great topic. You will learn everything you need to write Token Bucket and more!
You might also like
December 11, 2014
Summary: There are many great resources out there for learning Clojure. Some of the resources are paid, but fortunately many of the best resources are absolutely free.
Are you interested in learning Clojure? Would you like to get started without risking your money on a book? Is there something that can help you even know whether you want to learn Clojure?
Well, yes.
I learned Clojure for free. In fact, I'm a bit of a cheapskate. For many years I would avoid spending money if there was a free alternative. I also keep an eye on resources that pop up around the internet.
I have assembled what I believe to be the best resources to start with if you want to get into Clojure but want to spend $0.
Beginner
Start here if you have never programmed in Clojure before.
Reading
The ClojureBridge curriculum is a community effort that is informed by real-world workshops. It is excellent for starting.
A fun and adventurous guide through the Clojure journey.
A direct approach to bringing more people (especially underrepresented groups) to the Clojure community. The style of this book is very direct and approachable.
Exercises
For beginners, these two exercise sets will give you plenty of practice with the details of the language.
Exercises you can do right in your browser.
Exercises you do with git and an editor on your local machine.
Editors and Installation
The best install guide is from ClojureBridge. If you follow this completely you'll have Light Table installed, which is a great (and open source) editor that makes Clojure development very easy.
Documentation
An organized list of built-in Clojure functions and macros.
A searchable version of the cheatsheet with documentation, source code, and examples.
Workshops and Events
ClojureBridge hosts free workshops around the world to encourage diversity in the Clojure community. Go to one if you can.
There are many groups on Meetup.com that are interested in Clojure and functional programming. Find one in your area and get in touch.
Always hundreds of people on and someone to help you if you get stuck.
Clojure discussion group. This contains everything from announcements of new library releases to beginners asking questions.
Ask questions on stackoverflow. They are usually answered very quickly.
Videos
A great high-level introduction to the philosophy behind Clojure.
Once you've got the basics and need to go deeper.
Reading
An open source, community-built book (also available for purchase). It includes many real-world, practical "recipes".
Exercises
Once you're writing Clojure code, do these exercises and get comments on your style.
These exercises are not Clojure-specific. They are small problems but are usually tricky enough to require deep thinking. I cut my Clojure teeth on Project Euler before other systems existed.
Documentation
The official documentation.
A searchable community documentation project.
Cross-references between Clojure libraries. Very interesting stuff.
Editors
At this point, you may want to think about dedicating some time to learning a different editor. Emacs is the most popular, and many people use vim.
An Emacs plugin called CIDER is the most popular way to use Emacs with Clojure. It gives you lots of functionality.
Many people use vim, so if you like it, here's a setup guide.
Conclusions
There is a ton of value in these free resources. But if you'd like to venture into something that is tailored for bringing up up to speed in Clojure from nothing to writing code and you don't mind paying, check out LispCast Introduction to Clojure. It's 95 minutes of screencasts, exercises, and visuals and there's nothing else like it.
You might also like
March 26, 2015
Summary: Unit testing in Clojure is straightforward. Here are a few testing ideas as they apply to Clojure.
Most of the Unit Testing literature discusses how to unit test Object Oriented code. However, Unit Testing is very useful for functional code, including Clojure. I'd like to document a few unit testing ideas as they relate to Clojure.
The Unit you're testing is the function
Unit testing is about testing the smallest unit of code. In Object Oriented languages, the unit is the class. In functional languages, the unit is typically the function. This is true in Clojure. If you're testing individual functions, you're unit testing Clojure.
Example-based tests
The easiest kind of tests to do is example-based, which means you test that for a given argument, you get a known return value. Let's look at a simple example:
(deftest addition-tests
(is (= 5 (+ 3 2)))
(is (= 10 (+ 5 5))))
You're testing that + works on two different inputs! Notice: 2 lines and 2 inputs, it looks like we'll get linear growth in tests as our coverage increases.
Round-trip testing
Ok, it's not exactly unit testing, if you are strictly going by the definition of "unit", because you're actually testing two functions. But who's really being so strict? A really useful kind of test is the round-trip test.
In Clojure, pr-str prints a value readably, meaning if the value can be read back in, this will make a string that could be read in using clojure.edn/read-string. You can do a round-trip from value to string back to value, and the two values should be equal. You're testing the property that these two functions are inverses of each other.
Example:
(deftest pr-str-read-string-round-trip
(is (= [1 2 3] (read-string (pr-str [1 2 3]))))
(is (= {:a :b :c :d} (read-string (pr-str {:a :b :c :d})))))
Again, we're getting linear test growth.
Here's another example where I test that addition is the opposite of subtraction:
(deftest +---round-trip
(is (= 5 (-> 5 (+ 10) (- 10))))
(is (= 10 (-> 10 (+ 100) (- 100)))))
What examples to test
If you're writing example-based tests one-by-one, and you're getting linear benefit for your examples, you've really got to maximize what you test, because linear growth is actually quite bad. In that case, what do you test? The best thing to test are the corner cases. Corner cases are mostly domain-dependent, but there are some domain-independent ones.
Empty collections
It's good to test what happens when you call a function on an empty collection. It could be that you didn't handle that, or didn't handle it correctly. The biggest gotcha is stuff like dividing by the size of the collection. If it's empty, the size is zero, and that's undefined.
Empty strings
The bane of the web programmer's existence, empty strings are usually not valid input, but of course that doesn't stop someone from passing one in. Are you testing that it's valid?
Zero
Zero is actually a typical corner case. Try it out.
One
One is also a typical corner case.
Normal cases
You should have at least one normal case to test the expected behavior. A normal case is a list with 5 elements, or a small integer (7, 12, 34).
Bugs
Now, here's the thing that makes having a test system set up totally worth it: having a place to put test for known and fixed bugs. If someone reports a bug, it's really nice to reproduce it in code in a failing test before you fix it. If the bug happened once, it could happen again, so make it part of your anti-regression suite.
Multiple assertions on the return value
One last thing that happens in Clojure is you want to assert a few things about the same return value. Instead of calling the function several times, why not save the value and assert a few things about it?
(deftest map-test
(let [return (map - [1 2 3 4 5 6 7])]
(is (= 7 (count return)))
(is (every? neg? return))))
One step further
Ok, I've mentioned a few times that example-based testing does not scale. Code coverage grows linearly as the number of examples grows. How do you get better than linear? One way is to use property-based testing (also known as generative testing). Instead of the programmer giving examples, the program generates examples! Here's a preview:
(defspec edn-roundtrips 50
(prop/for-all [a gen/any]
(= a (-> a prn-str edn/read-string))))
This tests that any value (gen/any) can be printed to a string and read back in, and you get an equivalent value. Three lines. You can run this with as many randomly generated values as you'd like. Thousands. Millions. With three lines. That's leverage.
Conclusion
Ok, those are just a few ideas that could get you started with example-based unit testing in Clojure. If you'd like to start automated testing in Clojure, I suggest you check out LispCast Intro to clojure.test. clojure.test is the built-in, standard Clojure testing library that most systems use (or are compatible with). The LispCast course is an interactive course with animation, examples, screencasts, text, code samples, and more!
Now, if you'd like to up your game at testing, and want to get more than linear bang for your buck, you've got to get into generative testing.
You might also like
March 23, 2014
Summary: There are a number of web frameworks in Clojure, but beginners should roll their own server stack themselves to tap into the Ring ecosystem.
One question that I am asked a lot by beginners in Clojure is "What web framework should I use?" This is a good question. In Python, there's Django. In PHP, Drupal. And of course in Ruby, there's the king of all web frameworks, Ruby on Rails.
What framework should you use in Clojure? The question is actually kind of hard to answer. There are a number of web frameworks out there. Some people call Compojure a framework, though it is really a library. lib-noir does a lot of work for you. Then there's your true frameworks, like Pedestal or Hoplon, which provide infrastructure and abstractions for tackling web development. All of these projects are great, but for a beginner, I have to recommend building your own web stack starting with Ring.
Compojure is really just a routing library, not a framework. You can use it for your routing needs, though there are alternatives, such as playnice, bidi, Route One, and gudu. If you don't want to decide, use Compojure. It's widely used and works great. If you want to go in depth, read the docs for the others. They are each good for different cases.
lib-noir comes from Noir, which was a web framework (now deprecated). It was easy and provided a bit of plumbing already built for you, so you could just start a project with a lot of the infrastructure built in. lib-noir is that infrastructure in library form. I haven't used it, but a lot of people like it. However, when I look at it, I see that most of what it provides I either won't use or it is trivial to add myself. That would normally be ok if there was huge adoption for it (like with Rails) so you get an ecosystem effect, but there isn't. lib-noir is used but certainly not dominant.
Pedestal has a lot of backing. It aims to tackle single-page apps by providing a sane front-end environment using ClojureScript in the form of a message queue. If you're into "real-time apps", this may be for you. Though, I would caution you that it's not for a Clojure beginner. Pedestal introduces a lot of new concepts that even experienced Clojure programmers have to learn. The tutorial is long and arduous. You will have problems learning Pedestal without knowing Clojure.
Update: Pedestal has changed dramatically since I last looked at it. It is no longer a frontend framework. It is now a high-performance backend server for high performance asynchronous processing. It's worth looking at if you need that. Otherwise, I stick with my recommendation to use basic Ring.
Hoplon is also designed for web apps. It gives you a DOM written in ClojureScript (including custom components), dataflow programming (like a spreadsheet), and client-server communication. It's a bold step, but again, requires you to buy into programming models that will take a long time to understand. If you are not already familiar with Clojure, you are asking for trouble.
There are other frameworks out there. But I recommend rolling your own stack. If you're learning Clojure, the best way to grasp how web apps work is to get a Ring Jetty adapter set up with some basic handlers. Add existing middleware as you need it. Write some middleware of your own. Use Compojure to route. Use Hiccup to generate HTML. That setup will get you a long way.
Ring is just functions. With a few basic concepts and a copy of the Ring SPEC handy, you can build a web server very quickly that does exactly what you want and you can understand every aspect of it. The experience of building one yourself can teach you a lot about how the other frameworks are put together.
What's more, Ring is dominant. Most people write functionality (in the form of middleware and handlers) assuming Ring and no more. So by staying close to the metal, you are tapping into a huge resevoir of pre-written libraries that are all compatible with each other. Ring is the locus for the Clojure web ecosystem.
Wiring up your own middleware stack is not that daunting. If you want guidance, my Web Development in Clojure video series is now for sale. It starts with a brand new Clojure project and ends with a fully functional app, backed by a database, and hosted on Heroku (for free!). In one hour, it explains all the concepts and shows you lots of examples. There's lots of exercises to get your brain whirring.
Recommended stack:
Then keep adding and customizing.
You might also like
March 07, 2014
Summary: Macros should be avoided to the extent possible. There are three circumstances where they are required.
There's a common theme in Lisp that you should only use macros when you need them. It is very common to see a new lisper overuse macros. I did it myself when I first learned Lisp. They are very powerful and make you the king of syntax.
Clojure macros do have their uses, but why should you avoid them if possible? The principle reason is that macros are not first-class in Clojure. You cannot access them at runtime. You cannot pass them as an argument to a function, nor do any of the other powerful stuff you've come to love from functional programming. In short, macros are not functional programming (though they can make use of it).
A function, on the other hand, is a first-class value, and so is available for awesome functional programming constructs. You should prefer functions to macros.
That said, macros are still useful because there are things macros can do that functions cannot. What are the powers of a macro that are unavailable to any other construct in Clojure? If you need any of these abilities, write a macro.
1. The code has to run at compile time
There are just some things that need to happen at compile time. I recently wrote a macro that returns the hash of the current git commit so that the hash can be embedded in the ClojureScript compilation. This needs to be done at compile time because the script will be run somewhere else, where it cannot get the commit hash. Another example is performing expensive calculations at compile time as an optimization.
Example:
(defmacro build-time []
(str (java.util.Date.)))
The build-time macro returns a String representation of the time it is run.
Running code at compile time is not possible in anything other than macros.
2. You need access to unevaled arguments
Macros are useful for writing new, convenient syntactic constructs. And when we talk about syntax, we are typically talking about raw, unevaluated sexpressions.
Example:
(defmacro when
"Evaluates test. If logical true, evaluates body in an implicit do."
{:added "1.0"}
[test & body]
(list 'if test (cons 'do body)))
clojure.core/when is a syntactic sugar macro which transforms into an if with a do for a then and no else. The body should not be evaled before the test is checked.
Getting access to the unevaluated arguments is available by quoting (' or (quote ...)), but that is often unacceptable for syntactic constructs. Macros are the only way to do that.
3. You need to emit inline code
Sometimes calling a function is unacceptable. That call is either too expensive or is otherwise not the behavior you want.
For instance, in Javascript in the browser, you can call console.log('msg') to print out a message and the line number to the console. In ClojureScript, this becomes something like this: (.log js/console "msg"). Not convenient at all. My first thought was to create a function.
(defn log [msg]
(.log js/console msg))
This worked alright for printing the message, but the line numbers were all pointing to the same line: the body of the function! console.log records the line exactly where it is called, so it needs to be inline. I replaced it with a macro, which highlights its purpose as syntactic sugar.
Example:
(defmacro log [msg]
`(.log js/console ~msg))
The body replaces the call to log, so it is located where it is needed for the proper behavior.
If you need inline code, a macro is the only way.
Other considerations
Of course, any combination of these is also acceptable. And don't forget that although you might need a macro, macros are only available at compile time. So you should consider providing a function that does the same thing and then wrap it with a macro.
Conclusion
Macros are very powerful. Their power comes with a price: they are only available at compile time. Because of that, functions should be preferred to macros. The use of macros should be reserved for those special occasions when their power is needed.
You might also like
October 08, 2013
Way back in May 2012, Ambrose was beginning work on Typed Clojure for Google Summer of Code. I interviewed him to get an idea of what kind of type system we could expect. Since he's now running a very successful crowd funding campaign, I thought I'd bring up this blast from the past.
You might also like
September 11, 2013
I am pretty excited about this project. Please check it out.
You might also like
