Boris Kourtoukov works with wearables and other interesting media projects. The talk description says a lot:
This could not be achieved without Clojure's ability to inter-operate with its host platforms, provide tools for live prototyping, and the incredible diversity of interests within the Clojure community.
This post is one of a series called Pre-West Prep, which is also published by email. It's all about getting ready for the upcoming Clojure/West, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-West Prep is about. I want to enhance everyone's experience at the conference by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations.
Clojure/West is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/West or Cognitect. It is simply me (and helpers) curating and organizing public information about the conference.
Anthony Marcar's talk at Clojure/West is about how to build larger Clojure applications.
Background
Clojure was released in 2007. That means it's coming up on 8 years. It is also a powerful language for parallel computations, like we see in Cascalog and Storm. As big data has become more popular, Clojure's ease with processing lots of information has given it a great role to play recently. It came about at just the right time to take advantage of the big data boom. Several large companies acquired smaller Clojure companies, including Walmart, Staples, and Twitter. And because Clojure runs on the JVM, it's perfect for the enterprise. But we don't have many experience reports about how they have learned to use it.
Anthony Marcar works at Walmart using Clojure. I'm sure there will be some great information in this talk. I don't know specifically what it will be about, but it might be interesting to contrast it with Reflections on a real-world Clojure application, a talk by Malcolm Sparks.
This post is one of a series called Pre-West Prep, which is also published by email. It's all about getting ready for the upcoming Clojure/West, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-West Prep is about. I want to enhance everyone's experience at the conference by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations.
Clojure/West is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/West or Cognitect. It is simply me (and helpers) curating and organizing public information about the conference.
The main project tool in the Clojure community is Leiningen. In Leiningen, you write a project's configuration using a declarative syntax. In order to add build actions, you generally have to add plugins, which also need to be configured.
Boot takes a different approach, which is to code everything in Clojure. Boot is just a library and you script builds similar to how you might do so with shell scripts or Make, but you're in Clojure.
Boot is like the lisped-up lovechild of Git and Unix in that it provides abstractions that make it much more pleasant to write code that exists at the intersection of your operating system and your application.
Why it matters
Builds and deployments are getting more complex. A tool needs to be able to build a Docker container, push to Heroku, tag a release, spin up a server, etc. Can a tool built around declarative configuration and plugins keep up? The creators of Boot say the answer is "no". Boot's approach is different from Leiningen's, and so is not competing head-on, and yet still fills a need. Also, Boot's branding and developer pedigree seem to put it on firm foundation. There could be room for a second project tool. If there is, it will likely be Boot.
This post is one of a series called Pre-West Prep, which is also published by email. It's all about getting ready for the upcoming Clojure/West, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-West Prep is about. I want to enhance everyone's experience at the conference by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations.
Clojure/West is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/West or Cognitect. It is simply me (and helpers) curating and organizing public information about the conference.
Melanie Mitchell's keynote at Clojure/West is about building computer systems that recognize visual situations.
Background
Melanie Mitchell is a researcher in Artificial Intelligence, complex systems, and machine learning. She did her PhD with Douglas Hofstadter, which resulted in Copycat, which was featured prominently in Hofstadter's book Fluid Concepts and Creative Analogies. Perhaps coincidentally, Copycat was written in Lisp. The branch of Artificial Intelligence that she works in is probably different from what you would read in most texbooks or learn in college courses.
This post is one of a series called Pre-West Prep, which is also published by email. It's all about getting ready for the upcoming Clojure/West, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-West Prep is about. I want to enhance everyone's experience at the conference by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations.
Clojure/West is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/West or Cognitect. It is simply me (and helpers) curating and organizing public information about the conference.
People have been asking me how they can get ready for all of the great talks lined up for Clojure/West. There are many topics, and people are feeling overwhelmed. They might know about some of the ideas, but not all of them.
Wouldn't it be nice to do just a little homework before the conference so that you would be sure to maximize your understanding and be able to participate in the conversations afterward?
Unfortunately, googling any topic reveals lots of material, and it's not always the best. It would take hours to sift through the background material for just one topic.
That's why I am publishing the Pre-West Prep. Each day, I will send you one or two things to read or watch (whichever is better) to get up to speed on the topics of the talks (which are listed on the Clojure/West speakers page). I'm used to finding great content for the Clojure Gazette, so you can be sure it will be helpful.
Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media. That rich context is what Pre-West Prep is about. I want to enhance everyone's experience at the conference by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations.
There are 33 talks listed and not much time to publish all of them, so I will have to multiple ones per day. I've got Nola Stowe helping me write the summaries and interviews! If you'd like to help, !
Sign up below
Once you sign up, you'll start receiving the emails, starting with the first, one (or more) each day. It's a great way to get ready and excited for the conference.
Clojure/West is a two-track conference, which means often you must choose between two equally awesome talks. You will miss some talks. Preparation is key to having a great experience.
As you probably know, conferences are social events. The talks are mostly an excuse to get together, meet each other, and talk about Clojure. The conversations that happen after a talk can be very rewarding. But if you don't understand why the topic is important, you may feel left out. Don't let that happen! You're travelling far and paying (or expensing) lots for the ticket, hotel, and travel. Make the most of it!
Clojure/West is a conference organized and hosted by Cognitect. This email list is in no way official. It is not sponsored by nor affiliated with Clojure/West or Cognitect. It is simply me (and other helpers) curating and organizing public information about the conference.
Summary: Most technical writing is obtuse unless you already know the topic. Focus on the learner and their skills and it will clarify your writing.
Why is most technical material completely undecipherable unless you already understand it?
Oh, the joys of learning about monads:
All told, a monad in X is just a monoid in the category of endofunctors of X, with product × replaced by composition of endofunctors and unit set by the identity endofunctor.1
All jokes aside, this is typical of technical writing. Why is it so obtuse? I'm sure there are some people who could read this and confidently nod their heads. But to the rest of us, it's completely obtuse.
But obtuse language is not limited to mathematical texts. Here is an excerpt from a section of a Clojure book introducing integers in Clojure:
Integers comprise the entire number set, both positive and negative. Any number starting with an optional sign or digit followed exclusively by digits is considered and stored as some form of integer, although several different concrete types are used depending on the circumstance. Integers in Clojure can theoretically take an infinitely large value, but in practice the size is limited by the memory available.2
I program in Clojure every day. I probably write integers in Clojure every day. Yet I still have to reread this paragraph to make sure I understand it. Yes, it makes sense to me, since I already know the material. But this paragraph does not teach. It simply records technical facts. A new Clojure programmer would gain just as much from this as from the monad explanation above.
I don't mean to pick on any books in particular. This is pervasive in technical books from all fields. It's why most learners consider themselves self-learners. They did the hard work of reading and rereading this dense, obtuse, and unhelpful material, and rightfully feel that the bulk of the effort was on their end, not the author's. When they finally get it, they don't credit the author. They credit their own perseverance.
Is this fair? You do all the work of writing the book. How can you get the credit?
The answer is surprisingly simple. You should focus on skills, not technical knowledge. Reread the integer paragraph above. Notice that it's all about facts. Where does the reader fit in? What does the reader do with this knowledge? Why should the reader care? All of that is absent.
Without answering these questions, the paragraph is most generously described as reference material. It might contain knowledge you will one day need. It will sit on your shelf until you have an actual question to answer, then you'll look it up. And you'll feel like a self-learner. But it's not teaching you.
It's like a clock. When you need to know the time, you can read it. But it can't tell you why you should care that it's 11:14pm.
How do you focus on skills? Think of the poor reader! Ask yourself: what do they do in their real life? Let's go through the integer paragraph and convert it into skills. We're trying to convert the fact into something they might want to do with it.
Before I do that, though, let me say this: these are my interpretations of possible skills. Not all of these are totally relevant. I didn't write the paragraph, I'm just interpreting it. So a lot of it might seem pedantic or unimportant.
Skill: If you want to make a positive integer, type only digits.
Skill: If you want to make a negative integer, put a - in front of digits.
Skill: If you want to emphasize that an integer is positive, put a + in front of digits.
Skill: When you're at a party and you want to impress your Clojure friends, you can mention that Clojure represents integers with several concrete types.3
Skill: If you want to make really big integers, you can.
Skill: If you see an OutOfMemory error, check that you are not making an integer that is too big.4
It's good to break text down like this. Often it shows what skills are important and what should be saved for later (so late, sometimes, that they fall off the end of the book!). As the teacher, your job is to choose what they skills they need to learn.
Notice that this begins to look like a task analysis. Each of these skills should be broken down as far as they need to go for your audience. These raw skills that I wrote above are useful like this as part of the process, but they need to be refined a lot. That's the goal of making a curriculum, which is another post.5
Conclusion
Here's the hard truth: writing down facts is the easy part. It's what lazy authors do. There are thousands of technical books out there written by people that do no more than document someone's (sometimes impressive) knowledge. Teaching is hard, but not impossible. It will set you apart.
Learning is about skills. Even what seem like plain facts have a reason for existing. People want to do something with them. The difference between writing and teaching is that teaching focuses on the learner and their skills, whereas mere writing is just about the topic.
If you like the idea of teaching technical material (not just writing about it), please sign up for my Technical Teaching mailing list, where I'll occasionally share skills for making material that helps people learn.
From The Joy of Clojure, Second Edition, by Michael Fogus and Chris Houser, page 27.↩
Can you think of a better reason for mentioning this? Who needs this knowledge, and for what? It becomes clear in subsequent paragraphs.↩
Can you come up with a better reason to mention this?↩
Just for reference, I think this paragraph touches on the following skills: * Write positive and negative integers. * Read positive and negative integers. * List the concrete types that may represent an integer. * Recall that larger integers may take more space than smaller integers.↩
Summary: A common failure in distributed systems is a server with a rate limit or with no limit but begins failing due to load. A standard solution is to retry after waiting a small time, increasing that time after each failure. We create a macro to handle this waiting and retrying.
A few days ago I wrote about a high-level way of handing intermittent errors, particularly in a distributed system. The way was simplistic: when you get an error, try again, up to a few errors. A slightly more nuanced approach is to back off before you try again. Each time there's an error, you wait longer, until some maximum time is reached.
The problem
Let's say you're hitting a service with a rate limit. That rate limit could be enforced or implicit1. You've got lots of computers hitting it, and it's impossible to coordinate. No matter how hard you try to keep under that rate limit (and you should try), you will eventually break the limit. Retrying immediately when the server is too busy will actually make the problem worse. You will give it yet another request to deny. At the same time, it might be hard to distinguish "I'm too busy right now" from "I'm never going to recover".
The solution
I don't know what it's really called. I call it Exponential Backoff. It's also easy to turn into a separate routine:
(defn exponential-backoff [time rate max f]
(if (>= time max) ;; we're over budget, just call f
(f)
(try
(f)
(catch Throwable t
(Thread/sleep time)
(exponential-backoff f (* time rate) rate max)))))
This one has the same structure as try-n-times but will sleep before recursing. When it recurses, the time is multiplied by the rate. And when the last wait is more than the max, it will try one more time. Failures from that last try will propagate.
This will retry after waiting 1 second (1000 ms) the first time, then double it (the 2) each time. When it waits 10 seconds, it won't retry any more.
Slightly more useful
Ok, so I don't use this exactly. What I use is slightly more complicated. I've found that I often can tell if it's a rate limiting problem if I look at the exception. So, let's pass it a predicate to check.
(defn exponential-backoff [time rate max p? f]
(if (>= time max) ;; we're over budget, just call f
(f)
(try
(f)
(catch Throwable t
(if (p? t)
(do
(Thread/sleep time)
(exponential-backoff f (* time rate) rate max))
(throw t))))))
This one only recurses if the predicate returns true on the exception. Let's service mentions "queue capacity" in the body of the HTTP response when it's too busy:
Also, add it to your Clojure Emacs config for better formatting, because this one wants the args on the first line:
(put-clojure-indent 'try-backoff 1)
This tells Emacs to make the second argument ((println "trying!")) one indentation in, instead of directly under the first ([]).
Warning
All of the try3 warnings apply. The stuff you're doing inside needs to be idempotent!
Conclusion
This pattern is another cool, reusable component to help build reliability into a distributed system. Small, intermittent failures are pervasive. And a common form of error is a server being too busy. Being able to handle this type of error quickly and systematically is going to make your life easier.
Though Clojure does not have specific solutions to distributed systems problems, coding them up is short and straightforward. If you're interested in learning Clojure, I suggest you check out LispCast Introduction to Clojure. It's a video course that uses animation, storytelling, and exercises to install Clojure into your brain.
Summary: Solarized is an awesome color scheme that I use all the time. I was tired of looking up the hex codes, so I made my own cheat sheet. You can download it.
I like Solarized. I use it in Emacs, in code when I highlight, and even in my presentations. I'm not good at choosing colors. Ethan Schoonover has made a really great color scheme that's great for what I do.
Despite having used it for years, I still have trouble naming the colors. I also don't think I'll ever remember the hex codes for HTML/CSS. Also, some apps still don't let you select colors using the system color dialog. The most accurate way to enter them in is hex RGB. So today I got fed up and made this cheat sheet.
Summary: Object-oriented dispatch is contrasted with functional dispatch, but they are shown to be two one-dimensional views of the same two-dimensional data. Clojure does not provide the two-dimensional representation, but does interesting things to transcend the one-dimensional views.
About a month ago, I wrote a post about how OO-style is the dual of functional-style. OO focuses on the data first, while functional focuses on the code. It is cool that the correspondence between the two is fairly clear and mechanical. It means that they're equivalent in a way. The distinction is mostly important when choosing how to represent a problem. A language should provide ways to express both (as Clojure does), and a way to translate between the two (which Clojure does not).
But it goes further than data vs code. In OO, the object is the unit of data (which manifests as the principle of encapsulation). But the object is also the unit of dispatch. Objects know their class, and classes know the implementations of their methods. Class first, behavior second. In a functional style of programming, the function is the unit of dispatch. A function knows how to react to all possible classes of arguments. Behavior first, class second.
So again, we have a kind of duality. The classic way to explain this is to show it in a table.
method/class
String
Integer
Vector
...
toString
.
.
.
length
.
x
.
+
x
.
x
*
x
.
x
...
A dot represents an implementation, whereas an x represents an undefined operation.
In OO style, the class represents a single column. In functional style, the function represents a single row. Logically, however, the information is a two-dimensional table. That brings up a question: why don't languages store the information in this form internally? It is easy to project a column view or a row view from a table. And the x's in the table seem to be really useful for static checks.
Clojure does not represent its functions this way. But it does allow you to express your code in either a column view or a row view. The row view is the standard functional approach shown above. The column view is using deftype.
Example:
(deftype Person
Object ;; indicating I'm overriding methods from Object
(toString [p]
...)
java.util.Comparable
(compareTo [p]
...)
...)
Clojure goes a step further: instead of having to write out the entire column (all methods given a class) in one place, you can essentially index directly into the table given a method/class pair, and define the implementation directly. This only works with protocol methods (not methods on classes/interfaces due to limitations in the JVM).
OO-style and functional-style are duals in terms of dispatch. It's very related to the data-first or code-first duality I wrote about before. Clojure again straddles both sides of the duality and lets you write code in both styles, as well as transcend the column/row distinction entirely when using protocols.
How does your language deal with dispatch? Can you express the problem in the best way? If you're interested in this kind of topic, you would probably enjoy the Clojure Gazette. It's a weekly newsletter filled with content to inspire Clojure programmers. It's completely free and it's easy to unsubscribe.
Clojure pulls in ideas from many different languages and paradigms, and also from the broader world, including music and philosophy. The Clojure Gazette shares that vision and weaves a rich tapestry of ideas from the daily flow of library releases to the deep historical roots of computer science.
Summary: Many technical books skip very important skills you need to complete a task. To avoid skipping those skills, use task analysis to break a skill into steps.
This image reminds me of so many programming language books. The first chapter introduces the syntax, the second chapter is about writing a blog. What about all the stuff in the middle? I work hard in the LispCast courses to avoid skipping these steps.
The problem is clear: most technical skills are actually made up of hundreds if not thousands of smaller skills. I mean, tons of small decisions, perceptual skills, motor skills, etc. It's amazing really what people can chunk into a single idea. There are illustrators who think of the process as two steps. Likewise, many programmers are really good and think of complex code in two steps.
Our job as teachers is as clear as the problem: break down a skill into all of the smaller skills, then teach each of the smaller skills and how they lead to the big skill. It's so easy to state, and yet most people don't do this, even when they're writing for a beginner audience.
Why do so many programming language books spend so much time on syntax but fail at the intermediate stuff? It's because syntax is easy and usually already broken down for you. Once that's covered, it's hard to know the next step to teach if you haven't done the work of discovery. But they also focus on trivial stuff, like how to type literal integers. Hello? Really? If you bang on the number keys you can make integers.
So, how do you do it? How do real teachers do it? It's actually kind of boring. The kind of boring process that you might look at and say "oh, well, obviously". The process is simply to do it yourself and keep notes about every little thing you did. Ask yourself really specific questions like "Why did I start with two circles? Why do they overlap?"
How do I make use of it in LispCast? I always start with the code I want my learners to be able to write. I write it all out first. Then I start breaking it down. Every construct (if statement, library function, etc.) is obviously something to teach. Then there are the value-add things, like why I chose one way over another. What questions do I ask myself to make the decisions? All of these things can and should be steps.
Now, the next step is matching the steps to the learners. Maybe they already know something, so I can just mention it or if it's well known, just do it without mentioning it. Some things might still be difficult, so I might need to break them down further.
At the same time, it could be better to avoid a certain concept. Like if you'd normally do it with a cond but they already know if, perhaps you should stick to what they know so you don't have to add several minutes to explain something that's not relevant to your topic. I try to go deep, so it's important to minimize the breadth of the material in order to finish on time. All of these little skills go into a curriculum, which I'll talk about in a future post.
I don't think this process is hard or original. I learned it from a book about teaching. I do think that most people who write technical books might be subject-matter experts, but they don't have any training in teaching. Teaching is a huge advantage. Even a simple process like this could really differentiate you from the other technical authors out there. And breaking down a task tells you what you need to teach (no more writer's block).
If you like the idea of teaching technical material, please sign up for my Technical Teaching mailing list, where I'll occasionally share skills for making material that helps people learn.
