Ashton Kemerling: The path I took to Clojure is fairly roundabout. During my College years I was very interested in function programming, although ML & Haskell were my favorites back then. I think a large amount of my fascination with them stemmed from the fact that I had more experience than my classmates, my high school had a very rich set of programming courses, and I needed something interesting to play with while taking introduction to OOP courses.
Fast forward a few years and I ended up working at a job using Common Lisp professionally in a legacy web application. The application had a lot of problems due to the condition of CL libraries at the time and we were looking for something to switch to. Clojure was an obvious choice to look at. It's still a lisp, so a lot of our habits would still work, but it fixes the deployment & library issue that CL has while adding a much richer set of data structures. Unfortunately our application relied extensively on mutation (CL doesn't really encourage or discourage any coding style) and OOP, so a conversion was deemed too costly to attempt.
Over the course of the following years I continued to tinker with Clojure on the side. I was convinced it was a great way to do things, but I could never convince anyone to consider using it in their code base. Most places were either too invested emotionally in OOP, or had well tested and working codebases that made a rewrite unwise. So I mostly just hacked on side projects using Clojure over the intervening years without using it professionally. But in the last year or so I realized that Clojure makes a great secondary language for odd-tasks in non-Clojure codebases, which brings me to the current state of using Clojure to test a very large Ruby on Rails and Javascript application.
LC: Can you give a basic outline of how your Clojure program tested the web front end?
AK: We've basically tested in two ways. The first was to simply compile our JS file along with Clojurescript tests and run them in a PhantomJS process similarly to how you'd run Clojurescript or Javascript tests normally. We're still working on this approach, but it's slower to work on that I would have hoped.
The second way was to leverage the fact that Clojure's on the JVM. This involves mixing JDBC, HTTP libraries, and Selenium in novel and exciting ways. This has been the most fruitful way of testing, and what I'll be focusing the most on during my talk. We've used the code I've written as both a tool to hunt down API mistakes, and as a means of narrowing down the reproduction steps on complicated bugs reported by the user.
LC: Can you explain a little more about that? What do you mean by "narrowing down reproduction steps"?
AK: Sure. I'm testing Pivotal Tracker, a very large (~24kloc) Javascript "single page app". With large apps like this there are ways for the application to subtly fall out of sync with the server or process data in a degenerate way.
In particular a lot of times you'll get a report from a user that says something along the lines of "I reloaded and it crashed". You can dig up the logs from that session, but you'll get the last 20 or 30 steps they did before the crash, which doesn't really help you out a whole lot. So we occasionally turn the generative tests towards a type of action (in this case, do things and then refresh) in the hopes that it will help us narrow down to the minimal reproduction steps required to trigger a problem.
In the case that happened last week, we were able to find 2 distinct actions that would provoke the issue reliably after about 20 minutes of test modification and running. Obviously this took hours off of the process required to find the actual root of the bug, since 2 is orders of magnitude easier to work with than 7 or 10 when debugging.
LC: I see. So how is test.check able to reduce the reproduction steps?
AK: Test.Check provides "shrinking". Just as the generators used to create randomized data have the ability to produce more complicated data, they also have the ability to produce less complicated data. Test.Check records all of the failures it finds and attempts to simplify them and find the smallest failing case it can.
LC: A lot of the Clojure/conj participants will be new to Clojure. What resources would you recommend them to make the most of your talk?
AK: They can't go wrong with the test.check README or any of the blog posts mentioned therein. I recommend reading the source directly because it's heavily commented, in particular I recommend the generator source, because the generators represent 90%+ of the test.check API a user will interact with.
LC: If Clojure were stranded on a desert island, what one book would it bring?
AK: A boat building book, clearly.
LC: Awesome! Thanks for the interview, it was a pleasure.
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
Bozhidar Batsov: I’m very fond of Lisps and I love exploring new programming languages. In 2009 Common Lisp was my favourite Lisp dialect and I was often solving small programming exercises in it. I wanted to do some “real work” in CL, but the ANSI standard had left out basic things like networking, GUI, etc. and relying on distribution-specific extensions didn’t feel right to me. I was working as a Java developer at the time, so when I stumbled upon a news about the release of Clojure 1.0 I was super excited. A practical Lisp seemed like a dream come true to me! I immediately started playing with Clojure and enjoyed it immensely. I was impressed by the immutable data structures, the seamless Java interop and the powerful concurrency primitives. Along the way I developed a deep appreciation for the merits of functional programming and explored a few other functional programming languages.
LC: What is CIDER and how did you wind up as its maintainer?
BB: CIDER is an interactive Clojure development environment for Emacs. It’s pretty similar to SLIME (which I consider CIDER’s spiritual ancestor) for Common Lisp and Geiser for Scheme/Racket. People who are not familiar with Emacs should think of CIDER as a Clojure IDE plugin, that provides a ton of useful (Emacs-flavoured) features like interactive code evaluation, code completion, documentation lookup, code navigation and many others.
CIDER is built on top of the popular nREPL server and replaces SLIME + swank-clojure as the standard way to write Clojure programs in Emacs. A lot of the functionality in implemented in terms of reusable nREPL middleware, which makes the project pretty accessible to Clojurists who are afraid of Emacs Lisp. Portions of CIDER’s middlewares are used in other dev tools like vim’s fireplace and Gorilla REPL.
I was an early adopter of CIDER (which was named nrepl.el back then) as I was pretty excited to use a development environment tailored specifically for Clojure (unlike SLIME). The project felt pretty barebone (compared to SLIME) and my Emacs skill were pretty decent, so I started contributing features and improvements to it (I think the first major thing I contributed were the compilation error highlights). About one year after the project started, the original maintainer Tim King had to step back and handed over the maintenance to me. I’ve been hacking on CIDER pretty actively ever since and I have epic plans for its future! :-)
LC: What is the best resource for someone who wants to get started with CIDER?
While not strictly necessary, I’d encourage people to also have a look at cider-nrepl’s README.
A more extensive user guide (perhaps in texinfo format) and a few screencasts are on the roadmap, but I cannot promise when/if they’ll become a reality. Lots of bugs to squash and features to add prevent me from spending more time on improving the documentation.
LC: What is the best resource for someone (like me) who has migrated from SLIME and might not know all the cool stuff that's in there now?
BB: The README lists all prominent config options and commands (or at least I think it does). The changelog is also a great place to discover new features.
There’s a wiki entry on differences with SLIME, but it hasn’t been updated in a while, so people won’t learn much from it. Guess I should find some time to update it!
I’m always planning to write blog posts and do short screencasts for important new features, but unfortunately I never find the time to do so. Who knows, maybe next year...
LC: Where can people follow your adventures online?
BB: There are plenty of opportunities for people to stalk me online. :-) Here are some of the best options:
LC: One last question: If Clojure were a zoo animal, what would it be and why?
BB: A bear! Bears are (arguably) cute, smart and powerful. They are good at pretty much everything - they can run very fast, climb trees and swim. Does this remind you of a certain programming language?
LC: This was a great interview and I enjoyed it!
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
The next interview is with Colin Fleming. He is the developer of Cursive Clojure. His talk at Clojure/conj will be about that. Read the background to his talk.
Interview
LispCast: How did you get into Clojure?
Colin Fleming: I got into Clojure bit by bit, starting about four years ago. I was working in Java on highly concurrent transaction processing systems and the promise of a language that would make that easier was very appealing. I was also playing with Scheme at the time so it seemed like a good fit since I already knew the JVM really well.
I’ve used IntelliJ for Java for years now and I wanted to use it for Clojure too, so I installed La Clojure, which was the only option at the time. But I quickly got frustrated with its limitations and started fixing it up, porting the parts I worked on to Clojure as I went. That was really my first real Clojure project, which in retrospect wasn’t a great choice - it was a tough project to get started on. But it evolved into Cursive, which I’m very happy with, so it worked out well in the end.
LC: Developing an IDE is quite a big task. And the web site indicates that you will charge for it. What has the response been? How many users do you have in the Early Access Program?
CF: Right, it’s definitely a big task. Initially Cursive started as a hobby, just scratching my own itch. But I quickly realised that if I was going to develop all the functionality I wanted to add it was at least a full-time job, probably more. I had just taken some time off after leaving my previous job at Google and I thought I would see if there was appetite in the Clojure community for a professionally-supported development environment, and the initial response on the Clojure mailing list was pretty positive.
I released the first public beta version almost a year ago now, and the response has really been fantastic. Quite a few people have asked me when they’ll be able to pay for it so they can continue to support development, which is really nice to hear - I’m really happy people like Cursive so much. It’s hard to tell exactly how many people are using it, but each version tends to get around 10,000 downloads. There’s quite a long tail so it depends how long each version is out for, but generally there are around 2,000 downloads within 24 hours of making a release so I guess that’s roughly the number of people using it daily, or at least have it installed in the copy of IntelliJ that they use daily.
LC: I use Emacs but I can't really recommend it to someone who just wants to try Clojure. Does Cursive fit that bill and if so, how?
CF: Sure, I think Cursive is definitely easier to get started with than Emacs. In general IntelliJ’s interface is more like a standard application than Emacs, that familiarity helps a lot. And I try to make sure that Cursive is easy to get started with and to minimise the yak shaving required to open a Leiningen project and a REPL. There are still some outstanding things to do there, a big one has been that Cursive didn’t ship with default keybindings assigned - that’s fixed in the next release which should be out shortly. Cursive also has paredit turned on by default for Clojure which can confuse newbies, but it’s easy to toggle. And Cursive still doesn’t have a great story for a Clojurescript REPL, which is always a big pain point for people getting started - I’m hoping to fix this soon. Light Table really did a great job on that one.
Cursive has a user guide with a lot of screenshots that should hopefully make it easier for people to get started. It’s not 100% up to date with all the latest features, but it should help people to get going. I think having one definitive source of friendly documentation really helps at the beginning. I’d actually like to have an interactive tutorial in the IDE that people could run that would show them how to use the editor and possibly the basics of Clojure too, but that’s looking like a v2 feature at this point.
LC: I've always liked IDE's for their ability to smooth over the rough parts of a language. And tooling often trumps terseness in a language with less tooling. What are some of the trouble-spots you see in Clojure, from a beginner's perspective, that haven't been addressed with tools?
CF: The classic one that a lot of people seem to struggle with is the stack traces - I’ve never found them to be too difficult, but I worked with Java for years so I know what they look like. It’s easy to forget how difficult things like that are for people new to the JVM. Leiningen has done a great job of hiding things like the class path and dependency management as well, which can be really tough.
The other thing that I’d really love to see improved is the error messages. Since Clojure macro forms are essentially parsed by hand some of the error messages are really impenetrable. For example, something that might be a common newbie error like putting an (:import) clause ahead of the docstring in an ns form will give you a "java.lang.Character cannot be cast to clojure.lang.Named” error, and will also frequently attribute it to the wrong namespace - these sorts of errors can be really discouraging for someone trying to get started. Cursive internally uses something similar to the seqex library (from Chris Houser and Jonathan Clagget’s great Illuminated Macros talk) for its parsing, I’d love to see libraries like that being used in Clojure core.
Cursive doesn’t actually mark a lot of these errors in the editor yet, but I’d like to mark as many as possible. In particular, I’ll be able to mark things like function invocations with the wrong number of args, non-gensym’ed variables inside syntax-quote and many of the problems detected by tools like Kibit and Eastwood. I hope that’ll make Clojure less frustrating for newbies and more experienced folk alike.
LC: There's a bunch of stuff I'd like to ask you about. Let's talk about the practical side of developing an IDE. You say it's a plugin for IntelliJ. How much of the IntelliJ system are you able to leverage, how much is custom, and how much is a hack to fit the Clojure peg in the Java hole?
CF: Right, it’s currently only available as a plugin to IntelliJ. JetBrains actually put a lot of work into making their base platform reusable so I’ll be able to create a standalone IDE as well, as JetBrains have done with Redmine for Ruby, or PyCharm for Python. But it’s a reasonable amount of work and probably adds some support load with cross-platform issues so it hasn’t been a focus for me yet.
The IntelliJ infrastructure is really great - it’s certainly saved me a huge amount of work being able to build on top of it. It’s very impressive how well abstracted it is, I’ve encountered very few places where I’ve had serious roadblocks from IntelliJ itself due to the nature of Clojure. Most of the real difficulties are just intrinsic to how flexible the syntax of Clojure is, they’re not problems with IntelliJ. Obviously all the base IDE functionality comes for free (project management, VCS support, indexing infrastructure, debugger and so forth) and the Java support is very easy to hook into which is important for Clojure. The biggest part that’s totally custom is the symbol resolution, which generally has to be written for each language anyway.
Cursive also supports Clojurescript, and that’s been more challenging since the Javascript support in IntelliJ is only available in the paid Ultimate Edition and is closed source. Cursive currently implements fairly primitive Javascript support itself which is just enough for Clojurescript, but it would be nice to be able to integrate that better with the existing support - I’m hoping to in the future.
The part that is very different in Cursive from a more standard IntelliJ language plugin is that it all has to be extensible since everything in Clojure is based on macros, even core language features. People will eventually have to be able to add support for the libraries they use or for the forms they define in internal projects so Cursive is internally based around an extension API that I’ll open up at some point soon. More conventional languages generally have a fixed syntax, even if it’s very flexible like Ruby for example, so this isn’t an issue for them but it’s essential for Clojure.
LC: In the talk description, you mention Rename Refactoring. It's not something I can say I miss in Emacs, but perhaps I don't know what I'm missing. Do people make use of that and other refactorings?
CF: Oh yes, I’d definitely argue that you don’t know what you’re missing out on! The two operations I use a lot are rename and find usages - I’d struggle to live without them now, especially find usages. I also recently added refactorings to extract and inline let bindings, which are really useful - people definitely use them a lot. I’m planning to add support soon for extracting and inlining functions, as well as introducing parameters which I use a lot in Java. Refactoring in Clojure will never be as accurate as in Java, but it’s still very useful. Things like marking unused local variables is also something which sounds trivial but really catches a lot of bugs while you’re editing.
I think this is the main difference between something like IntelliJ and something like Emacs, actually. When I’m working on Java code in IntelliJ the refactorings and all the small operations they provide as intentions are so good that I really feel like I’m working directly on my program, not on text. We’ve all experienced that flow state when programming when you’re totally absorbed in what you’re doing and you lose track of time, whether you’ve eaten or need to go to the bathroom or whatever. IntelliJ makes it much easier for me to achieve that state because it provides a huge number of code manipulations that directly reflect what I want to do to my program, without me then having to muck around with text. I’ve seen comments online along the lines of “IntelliJ is great but I type so much faster in Emacs/vi/whatever”. I’d argue that approach is totally wrong - I don’t want to avoid typing because I’m lazy, I want to avoid typing because text manipulation is the wrong level of abstraction. Cursive is a long way from providing that experience, partly because of Clojure’s language characteristics and partly because it took teams of dozens of smart people at JetBrains years to achieve it for Java, but that’s always my ultimate aim whether or not I ever actually get there.
LC: What resources would you recommend people make use of before your talk in order to get the most out of it? Anything useful about refactorings or IntelliJ?
CF: I don’t think so, no - hopefully it’ll be pretty self-contained. I’ll be showing the features I’m talking about and explaining some of the implementation, so hopefully it’ll be understandable to everyone. And hopefully it’ll be interesting for everyone too!
LC: Where can people follow your adventures online?
CF: There’s a twitter feed for Cursive and there are also some mailing lists for general Cursive discussion. I’ve also been hanging out more in the #clojure channel on IRC as cfleming in case anyone has questions. I’ve been meaning to blog more about Cursive but so far that’s just one to-do item among many!
LC: Nice. One more question: what is Clojure's favorite breakfast?
CF: Thanks Eric, that was fun! I’m looking forward to seeing it, and to catching up at the conj too!
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
David Pick was kind enough to answer a few questions about himself and his upcoming Clojure/conj talk. You may want to read the background before you begin.
Interview
LispCast: How did you get into Clojure?
David Pick: A friend of my mine suggested I read Purely Functional Data Structures and told me the work in the book was the basis for Clojure's persistent data structures. I really enjoyed it and decided to give Clojure a try.
LC: Braintree must process a lot of very important data, since it's a payments service. What were you using before you developed the Clojure and Kafka system?
DP: I'd say for 98% of our systems we still use Ruby and Postgres, we're simply using Clojure and Kafka as a messaging platform to move data between systems.
LC: What is Kafka?
DP: Kafka is a system for doing publish subscribe messaging, that was built by LinkedIn to have extremely high throughput and replay ability of messages. It stores all messages sent through the system for a rolling window. So consumers simply say give me messages starting with offset n or whatever your most recent message is. This way if a receiving system goes down the consumer can just rewind the stream and replay messages that were missed.
LC: Could you describe the problem you were trying to solve? What were the requirements?
DP: There were actually two problems we were trying to solve. The first was to pull our search tool into a separate system that was more scalable than our current one (the current one just runs sql queries on main database). The second was to improve the accuracy and speed that our data warehouse is updated.
For the infrastructure part of the solution we ended up using Kafka, Zookeeper, and Elasticsearch which are all JVM based so that ended up heavily influencing our decision to use Clojure.
LC: You mentioned in the talk description that Clojure had certain advantages. What tradeoffs did Clojure, or your choice of technologies, bring about?
DP: The biggest tradeoff I think we made with Clojure is familiarity, I was the only one on the team who knew Clojure prior to starting this project. So that has certainly slowed us down. Other than that things have gone surprisingly well.
LC: I'm really fascinated by the engineering efforts to build very reliable and robust systems. Are you going to talk about that?
DP: Yep, that's the plan!
LC: Awesome.
Do you have any resources that would help someone new to Clojure or new to Kafka so that they can make the most of your talk? Some pre-reading or pre-watching materials?
LC: Where can people follow your adventures online?
DP: Twitter is probably the best place @davidpick.
LC: Who would win in a no-holds-barred fight to the finish, Clojure with an Arduino-controlled laser or a T-Rex?
DP: If Rich is writing the Clojure I give it a fighting chance, otherwise it's the T-Rex for sure.
LC: Awesome! Thanks for a great interview. I look forward to seeing you at the Conj.
DP: Awesome, thanks Eric!
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
Jeanine Adkisson: Through work, actually. I'd seen clojure around and studied some of its features from a language design perspective, but I've never really been a lisper, so it hadn't occurred to me to actually use it until we started a migration to datomic at GoodGuide. I admit to being a bit skeptical - I often find that lisps are a bit oversold - but a coworker of mine (@bonkydog) started really gushing about generative testing, core.typed, and the awesomeness that is leiningen, so I decided to give it a closer look.
LC: Your talk is about Variants. Can you explain Variants and how they are useful?
JA: Sure! Variants are a functional design pattern that should be familiar to anyone who's worked in the Haskell/ML family of languages (and to a lesser extent, Erlang). Their main purpose is to provide a structured way to manage polymorphism. In lisps and other single-type languages, they're not that widespread because we don't have to convince a type-checker that our polymorphism works out in the end. But we still have to convice users of our libraries and our doc writers that our functions make sense!
The method of representing variants I'll be talking about came out of some frustrations I was having at porting part of a language over to clojure, and finding it difficult to represent and destructure the trees I needed to process. But I was really impressed at how quickly David Nolen and Ambrose BS got back to me and shipped patches that made this approach possible (and typeable!).
LC: You mean David Nolen patched ClojureScript and Ambrose BS patched core.typed?
I wonder what it means about dynamic languages like Clojure that you can add Variants as a library instead of making it an integral part like in Haskell. What do you think about that?
JA: Yeah, actually it was a bugfix the match macro and the way that core.typed handled vectors that were key. What's even more interesting about variants is that there is really no library to write - all the machinery is already there. The solution isn't even particularly clever or novel - it's already in use in, for example, instaparse. It's just not on people's radars, or they have misconceptions about the approach from their OO training.
I love the fact that you can backport this pattern into a language that's not explicitly built for it. I wouldn't necessarily count that as a strength for dynamic languages in general though - I find a language's strengths often come from ways in which semantics are restricted. Clojure is already a great example of this because it explicitly rejects the use of willy-nilly mutable structures, to great benefit. Sure, you can program that way in Ruby, but the tools are already there in Clojure.
What I find coolest about the Clojure case is the extent to which core.typed has been able to backport types onto a mostly untyped ecosystem. The general problems with dynamic languages are still there, but they have to be made explicit with union types. How many functions surprised you by outputting (U <thing> (Value nil))? But because it's powerful enough to support variants in a lightweight and usable way, I think we now have a tool to deal with one of the most common culprits of surprise nils.
LC: I think the approach is worthy of study. I flipflop between statically typed vs dynamically typed because there are obvious problems with both. How close does your core.match and core.typed solution come to a full-on ML/Haskell-style, integrated solution?
JA: It's pretty comparable, and actually has some advantages. Since it's just vectors and keywords, it's trivial to handle in a generic way, and as long as you're not putting functions or host objects in them it's trivial to serialize and send over the wire. (Haskell and ML can do most of this programming for you with various generic programming extensions, but it's not straightforward or obvious). But yeah, core.typed is capable of refining types based on the tag matching code generated by core.match - a little roundabout, but it gets the job done. The match macro can also match against more than one object at a time, something OO languages constantly struggle with. There's a little more macro work to be done in core.typed to make it really straightforward to do variants, but it's a relatively trivial syntactic transformation.
In terms of typed vs. untyped, I think there's lessons to be learned from both. My instinct is to say that we just haven't really figured out how to make typed systems all that usable. Haskell and ML went a long way towards this, and I think they kind of hit a wall because of their restriction that compilation be decidable. In other words, the compiler is not allowed to infinite-loop - which sounds like a good idea until you line up all the restrictions required to make that happen. In dynamic languages, we don't have a semantic separation between "compile-time" and "run-time", but we still talk about "load-time" vs "run-time". And at load-time you can wield the full turing-complete power of the language. That's why I'm also excited about projects like Agda and Idris, which, while still statically typed and compiled, blur the lines between runtime and compile time - moving away from strict decidability guarantees in exchange for a more expressive system.
LC: A lot of the audience is going to be new to Clojure or they have never heard of variants. Is there a resource that you think would help make the most of their time? Just a little pre-reading or pre-watching so they feel confident with the material?
JA: Yeah, there's unfortunately not a whole lot targeting clojure or other lisps, which is one of the reasons I wanted to give this talk. There's a great section in the ocaml tutorial about them, and Learn You a Haskell covers them as well (although they call them ADTs). The approach I'll be talking about is most like what's used in Erlang, as in the temperature example here.
LC: Where can people follow your adventures online?
JA: Probably the best way to keep up with me is on twitter at @jneen_ (note the final underscore), or on github as @jneen.
LC: One last question: If Clojure were a comic book superhero, what would its superpower be?
JA: Hahaha, I'd probably say a shapeshifter, like most lisps I suppose.
LC: Thanks for an excellent interview. I had fun.
JA: Awesome, thanks!
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
Julian Gamble: It was about 2007, I was a Java Programmer toying with Rails. I was reading Steve Yegge and his amazing but incendiary blog posts. These had a big theme on languages and expressiveness.
I got curious enough to play with Common Lisp and Racket. I eventually worked through The Little Lisper - and found it hugely inspiring. I also read SICP and Paradigms of Artificial Intelligence Programming and was amazed (coming from Java) at what was possible with functional programming and homoiconicity.
In 2008 I was looking for a way to combine the power of Lisp and my Java background. I had a look at ABCL and Kawa and found they didn't really have much of a community behind them.
Then the Rich Hickey videos started coming out, "Clojure for Java Programmers" etc. Then lots of Java/Lisp people started coming out of the woodwork. I was fascinated by the design choices made and how opinionated the language was. I was delighted by the community that seemed to have come from thin air.
In 2009 Stu Halloway's Programming Clojure book came out and I devoured it. Then I went back and worked through the Little Lisper chapters in Clojure - and found functional programming had a lot more to it than a first-year course for students in the 80s. This meant idiomatic functional programming didn't work like that anymore. (Although the idea of building a Lisp from a few primitives is still very elegant).
In 2011 a Clojure User group started up in Sydney run out of Thoughtworks. This was a fantastic chance to meet up with people who programmed and developed their skills just for the joy of it - rather than just to pay a mortgage.
In 2013 Dave Thomas started running LambdaJam in Australia, and it meant that the functional programming community because more social and more willing to look at each others ideas across languages. It also meant we developed a shared appreciation for the heritage of functional programming. We also developed a deeper appreciation of academic ideas and their linkage back to our practice.
So now in 2014 when Rich Hickey gives a talk on Transducers, and explains that folds are fundamental - I go and read the linked papers from Hutton and Bird. Then I can go and talk about this with my local community of Clojure people. This is something that I wouldn't have done two years ago.
Clojure has been a great ride!
LC: In your talk description, you mention that you will relate core.async to the concurrency semantics we already know. Can you explain this briefly?
JG: The assumption here is that most Clojure people have a Java or C# background. From this background they would have an understanding of how to put process on a thread, similar to how we might use a future in Clojure. Alternatively, Clojure people may have seen how we can pass a function to an agent in Clojure.
Now it is true that core.async go blocks have some very specific queue sleep/wait semantics (CSP) that make it different to other concurrency models. In addition, core.async in JavaScript runs as part of the single-thread model of JavaScript, so it isn't semantically a thread, more of a 'process'.
Given those assumptions, the primitives we find for doing concurrency in core.async are very similar to the concurrency models we have seen in the past in OO languages or in Clojure.
In this talk we'll take a look at an example of taking a well known graphical example in Clojure, port it to ClojureScript and core.async. What we find is that many of the Clojure concurrency paradigms have a natural analogue in ClojureScript and core.async.
LC: Are there any topics to read up on or resources to look at that could help a beginner make the most of your talk?
Or they can send me an email @ and ask me what I'm up to.
(I've also been known to present at clj-syd on meetup.com - but that is probably only of interest to Sydney readers).
LC: One last question: If Clojure could eat, what would be its favorite food?
JG: Well I'm a huge fan of Dan Friedman's work on The Little Lisper/Schemer - which has frequent food references, such as a peanut butter and jelly sandwich. The book will at random reserve an empty page and state:
THIS SPACE RESERVED FOR JELLY STAINS!
So the answer would have to be PBJ sandwich!
(But for all the British, European, Australian and New Zealand Readers - we all know it as peanut butter and strawberry jam - and some might be puzzled by the combination of these two condiments, but would still appreciate Friedman's book.)
LC: Thanks for a great interview!
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
I got in touch with Lucas Cavalcanti. He is giving a talk together at Clojure/conj with Edward Wible, who was unavailable. The background to their talk is available, if you like.
Interview
LispCast: How did you get into Clojure?
Lucas Cavalcanti: I've got into Clojure at Nubank, the company I currently work for, about a year ago. Nubank was already using Clojure, so I learnt it as we evolved the project. I was already familiar with Lisp syntax and knew functional programming very well, so the move to Clojure was not too hard.
LispCast: Can you explain a little about Nubank? And how do you use Datomic?
LC: Nubank is a tech company for financial services that just went public:
We use datomic as our primary database system, taking advantage of the free datomic transaction logs for auditing, and adding more data to improve auditing. We also use the datomic query language (based on datalog) to analyze data, often joining multiple databases, since we have a SOA with each service having its own database.
We also use some datomic capabilities to do database sync with mobile clients and cache management.
LispCast: Your talk's description mentions some "hidden superpowers of Datomic". You mention that the solutions to these problems are generic. Does that mean they can work for any problem?
LC: The solutions we'll provide will have some small assumptions to work, but these assumptions don't depend on the entities types we're saving, just something about the database structure. Thus we said the solutions are generic, and could work for any problem that matches these assumptions.
Also means that we can write just one function that will serve to implement one of that "superpowers" for all entities/endpoints of your system.
LispCast: So a lot of people are new to Clojure, and certainly to Datomic. Can you explain Datomic briefly for those of us who have never used it?
LC: If you know Git or any version control system, Datomic brings to a database many powers you have with Git: Think of each database transaction as a commit that generates another version of the database. So you can get the database value at any point in time, check the changes made by a specific transaction, all the history of a certain entity or even create "branches" of the database.
It works by collecting facts like "the entity 123 has the attribute :speaker/name with value "Lucas Cavalcanti"" and gives a query language based on datalog that, once you get the basic concepts, makes much more sense than any other query language I've worked with.
LispCast: So, is there anything else you think people should read up on to make the most of your talk?
LispCast: Awesome! Thanks for a great interview. It was informative.
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
Paul deGrandis: I was a very early Clojure adopter. I came from Python (and before that a mix of C/C++/Java/Python), but I had always used Common Lisp to prototype ideas. Once I saw Clojure publicly announced, I gave it a shot and instantly identified with the Clojure Trinity: Simplicity, Power, and Focus. I began using it in all of my proof-of-concept work, and as I became confident in the language, integration with the platform, and my own skill set, I also started putting it into production.
LC: Can you define Data-Driven systems briefly?
PdG: Data-driven systems capture their interactions and operations as values - they are one possible conclusion of programming with values. We see this done in Datomic - queries are expressed as data. This concept is easily mapped to other systems - services, client-side applications, and more. Note that is more than just, "data all the things," this is, "data all the thing's things." Another example: Pedestal routes are represented as data ("data all the things"), but imagine if we captured the expression of an entire service as data. Imagine the properties that naturally fall out from that design decision. My talk walks through a real project that pushed this design decision as far as possible, and the steps that other developers can follow to do the same for their own systems.
LC: At Cognitect, they talk about Data > Functions > Macros. Is that what your talk is about?
PdG: My talk, at a very high level is about "Data > Functions > Macros," but it's mostly about modeling entire systems (not just interfaces) as data, why you'd want to, and why it's hard. It definitely challenges the previous notions (established by existing tooling) of what the conclusions of "programming with values" actually are (or could/should be). Tangentially, I'll talk about DSLs, but only in regards to constraining your domain as a means to increase expressiveness.
LC: I'm a big fan of working with data. But I've also seen its dark side. I've seen it pushed too far, or in the wrong area, to where it would have been better to use code. Have you noticed this? How do you avoid that?
PdG: I have indeed noticed it and it's a topic within my talk. Two crucial pieces of any great design (in software or otherwise) are the external constraints imposed by the problem, and the internal constraints imposed by the designer. Both are important, but it seems common that people overlook self-imposing constraints - and one needs them to achieve a really great design. I don't necessarily agree that capturing a system in data can be pushed too far, but I think it can be done without constraint or design intent - which produces a useless system. The crux is knowing that limitation and working within it. Outside of those limitations and constraints, one should fall back to foundation - regular Clojure code, etc.
LC: What is one thing you want someone to be able to do after they watch your talk?
PdG: I want someone to come away with a set of techniques to envision, constrain, and design data-driven systems. I want to create a spark that encourages Clojure developers to look beyond the initial merits of programming with values, and see further-reaching impact within their own systems - how to turn our ecosystem into a set of super powers.
I hope to provide a compelling example of how this approach plays out in production systems, and the general metrics and project impact it has.
LC: What resources would you recommend to a beginner so they could maximize their understanding of your talk?
PdG: My talk touches upon a lot of themes covered in some inspiring and informative talks: Tim Ewald's keynote from last year's Conj (Programming with Hand Tools), Rich Hickey's GOTO conference Keynote (The value of values), and previous Conj talks. I'm still searching for literature and books that I like on the topic, but I haven't hit any that I'd strongly recommend.
LC: Where can people follow your adventures online?
LC: If Clojure rode an animal, what animal would it ride?
PdG: A very serious pony.
LC: Thanks for a great interview. It was informative.
PdG: Thanks a ton! I really appreciate all of this!
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
I had a conversation about generating test.check generators with Steve Miner. He's giving a talk at Clojure/conj about that. Read the background to his talk.
Interview
LispCast: How did you get into Clojure?
Steve Miner: I think it was around 2008 when Paul Graham announced his new programming language, Arc. I was reading about Arc, when I came across a comment about Clojure and decided to take a look. I was very impressed with Rich Hickey's intro video. The immutability and concurrency features really resonated with what I was hoping to find in a new language. The Java integration made Clojure a practical language with a huge eco-system of tools and libraries. I was using Java at work and I managed to do a bit of Clojure for a side project, but mostly I was just dabbling. A couple of years later, I decided to work on my own with Clojure full-time.
LC: Can you describe Herbert for those who have never used it? Why would someone be interested?
SM: Herbert is a schema language for edn. The goal is to have a convenient language for describing the shape of Clojure data. I started out with an informal notation that I used for my internal documentation. For example, describing a map with certain required keys and the corresponding value types. I think you can guess what {:name str :num int} means as a schema. It turned out that with a little work, that informal notation could be used as a pattern language with a simple API for testing conformance. A Herbert schema is itself just edn data, open to all your Clojure tools. More recently, I added the ability to generate test.check generators from Herbert schemas, which makes it easy to generate test data.
LC: That sounds nice. Would you mind explaining test.check generators a bit for those who don't know?
SM: Reid Draper ported QuickCheck from Haskell to Clojure, and it became a contrib library called test.check.
Test.check is a property-based testing tool. A property is basically an invariant that should hold true over a range of input values. The test.check library gives you combinators that allow you to define generator functions which create data of specified types with optional constraints. The idea is to think about the whole range of possible inputs that your system should handle. Test.check then automatically tests across a random sample of generated data, probably generating example data that you might not have considered in your typical unit testing. If it finds a failure case where the desired property does not hold, test.check is smart enough to regenerate test cases so as to shrink the failure example to a reasonable size. That helps you isolate the cause of the failure.
LC: So you're able to automatically create generators from your schemas, which can also be used as contracts on your function arguments. Has having both improved your bug rate?
SM: I don't have any numbers, but subjectively I think it's helped. For me, Herbert schemas are primarily documentation tools, which help me to keep track of my data. That being said, I often test schema conformance in preconditions or asserts, especially with new code or when I'm trying to debug a problem.
Of course, I still make errors in specifying schemas and sometimes my properties aren't exactly correct the first time. Particularly with new property-based tests, I have to look carefully at failures in case the bug is actually in the test. My hope is that schema-based generators will make property-based testing easier to use.
Using test.check definitely improves my confidence that I'm finding bugs in testing and avoiding regression errors. It's been a great way to catch bugs in my own Herbert library.
LC: So you have runtime check to make sure the function arguments conform to certain schemas. And you have a generative test that exercises a large space of that schema. Sounds pretty good to me!
But it sounds like you're saying the primary benefit is more for you or other readers of your code. Can you elaborate on that?
SM: My approach started with a notation designed to help me keep Clojure data structures straight in my mind. I wanted something simple and terse, what I called a "whiteboard compatible" notation. My goal was that the notation should look something like the data it was supposed to represent as opposed to code or a type system.
So I began with documentation in mind, and I still think of that as the primary benefit. Once I got the idea of implementing conformance testing against formal schemas, the project became more about the code.
LC: In what other areas do you see schemas playing a part? The first thing that comes to mind is writing core.typed type annotations. Anything else?
SM: There's some conceptual overlap between schemas and type systems, but I see core.typed as a much more ambitious project. Herbert schemas only cover edn data and don't deal with function types, for example. The Datomic database naturally has a schema language, so it would be interesting to see if Herbert could useful for data modeling. In the near term, I plan to extend Herbert so that it supports the Transit datatypes.
LC: What resources would you recommend to a beginner who wanted to make the most of your talk?
LC: One last question: If Clojure were a food, what would it be?
SM: I'll say "pizza" because it's a food that is made by composing fancy toppings on a classic crust. And hackers like it while coding.
LC: Thanks, Steve, for a great interview.
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
Steven Yi: I think I came across it online about two years ago when it started to get a bit of a buzz. I had long wanted to learn a Lisp, but Common Lisp and Scheme never really clicked when I tried. Clojure was different though, and since I do a fair amount of Java programming, the possibility to integrate Clojure with my JVM music software Blue helped motivate me to get into it.
LC: Can you briefly describe Blue, Pink, and Score?
SY: Blue is a music composition environment, written in Java using the Netbeans RCP. It is a graphical environment with a score timeline, graphical instruments, mixer and effects, and more. It uses Csound as its audio engine and signal processing system. (Disclosure: I'm also one of the core developers of Csound.)
Pink is a Clojure library for building music systems. It features a low-level audio engine, higher-order events, and includes signal generation and processing functions.
Score is a Clojure library for higher-level music work, particularly for generating notes and scores using various strategies. It is designed to be generic and usable with multiple backends. Using Score and Pink together can be particularly effective and expressive, having one language for the full musical stack.
LC: Overtone has been getting a lot of attention lately. How do Pink and Score compare to Overtone?
SY:Ah yes, certainly the elephant in the room. :) I'll certainly address comparisons to Overtone in more detail during the talk. In a nutshell, Overtone builds upon SuperCollider. Of those two, Pink would compare more to SuperCollider than to Overtone, as it is concerned with low-level audio engine design. Score deals with higher-level abstractions in music, many of which are covered in Overtone. However, because of the full-stack Clojure design between Pink and Score, Score can interact with Pink in ways Overtone/SuperCollider can not. Pink/Score are currently not as mature nor as performant as Overtone/SuperCollider. There's a lot of overlaps between the various systems, and use cases and requirements for a user's project will guide them to one system or the other.
Hopefully that's enough to get going! I think the breakdown of music systems design I'm planning for the talk should hopefully make things clear.
LC: Is there any resource that could help someone new to Clojure, or new to music systems, get the most out of your talk?
SY: think it's a tough question about music systems as I don't know of any resources out there that really introduce the subject. Having some experience with Overtone would certainly be useful. I can say I'll probably mention SuperCollider, Csound, Common Lisp Music, Nyquist, and Common Music during the talk:
LC: Thanks, Steven, for a great interview! It was a pleasure.
SY: And thank you too!
This post is one of a series called Pre-conj Prep, which originally was published by email. It's all about getting ready for the upcoming Clojure/conj, organized by Cognitect. Conferences are ongoing conversations and explorations. Speakers discuss trends, best practices, and the future by drawing on the rich context built up in past conferences and other media.
That rich context is what Pre-conj Prep is about. I want to enhance everyone's experience at the conj by surfacing that context. With just a little homework, we can be better prepared to understand and enjoy the talks and the hallway conversations, as well as the beautiful venue and city of Washington, DC.
Clojure/conj is a conference organized and hosted by Cognitect. This information is in no way official. It is not sponsored by nor affiliated with Clojure/conj or Cognitect. It is simply me curating and organizing public information about the conference.
