dcug – Danish Clojure Users Group

Rich Hickey, creator of Clojure:

As should be obvious, Clojure is a labor of love on my part. Started as a self-funded sabbatical project, Clojure has come to occupy me far more than full-time. However, Clojure does not have institutional or corporate sponsorship, and was not, and is not, the by-product of another profitable endeavor. I have borne the costs of developing Clojure myself, but 2009 is the last year I, or my family, can bear that.

It would be such a shame if development were to slow down or stop completely because of such a small amount of money… So, please donate to the project.

See details,

News: I am giving a Clojure talk at Daimi, the computer science department at University of Aarhus. The talk is this coming Monday, 3pm. Location: CS department, IT-Parken, Aabogade 34, Aarhus (somewhere in the Computer Science department.) The talk is one hour, covering the fundamentals of Clojure, and focusing on the programming and concurrency model. It is sooo hard to give a proper introduction to Clojure in just one hour; there is really just so much to say!

Activity: It is not much activity we’ve been seeing here recently. How is Clojure use coming around in Denmark? Are you using Clojure for something? I really wish I could use it at work, or in some larger project, but of course it is not something you can push on clients… So for now it will be little pet projects.

Would one of you like to speak at the next DCUG meetup? It could be any Clojure-related topic, example code or project; anything, really. Let me know.

/Karl

Don’t forget the free dcug event at JAOO Aarhus 2009. Registration deadline is Monday, Sept. 28th (although it is also possible to sign up during the conference, logistics are easier if you sign up now!)

Ideas for agenda are welcome: Rich Hickey is attending and I expect he will give a talk on any topic we’d like!

Registration:

https://secure.trifork.com/aarhus-2009/freeevent/register.m?eventOID=2093

See you!

In the break at the Clojure talk in Copenhagen, I had a discussion with a bright young guy (I didn’t catch your name, sorry – but here is your chance to write at dcug ). He reminded me that while Clojure’s STM is very useful it does permit the “write skew anomaly” that developers need to understand. There is a description on wikipedia which I’ll steal an example from.

… imagine V1 and V2 are two balances held by a single person, Phil. The bank will allow either V1 or V2 to run a deficit, provided the total held in both is never negative (i.e. V1 + V2 ≥ 0). Both balances are currently $100. Phil initiates two transactions concurrently, T1 withdrawing $200 from V1, and T2 withdrawing $200 from V2.

If the database guaranteed serializable transactions, the simplest way of coding T1 is to deduct $200 from V1, and then verify that V1 + V2 ≥ 0 still holds, aborting if not. T2 similarly deducts $200 from V2 and then verifies V1 + V2 ≥ 0. Since the transactions must serialize, either T1 happens first, leaving V1 = -$100, V2 = $100, and preventing T2 from succeeding (since V1 + (V2 – $200) is now -$200), or T2 happens first and similarly prevents T1 from committing.

Under snapshot isolation, however, T1 and T2 operate on private snapshots of the database: each deducts $200 from an account, and then verifies that the new total is zero, using the other account value that held when the snapshot was taken. Since neither update conflicts, both commit successfully, leaving V1 = V2 = -$100, and V1 + V2 = -$200.

This may happen in Clojure. As wikipedia mentions the user can avoid write-skew by promoting one of the reads to a write (e.g., write to both of the refs in one of the transactions). This produces a write-conflict, which causes retry.

However, Clojure also has a function, ensure, which can be used in situations where write-skew is a problem (which is usually when you have some sort of constraints/invariants). The ensure function lets the user avoid “write skew” by “[protecting] … the ref from modification by other transactions.” (doc ensure). According to the documentation ensure allows more concurrency than promoting to a write transaction (but I haven’t read the source on this aspect, yet).

Comments welcome!

Note: This post is going to be a cliffhanger, because I have only just started building the system I’m describing here.

Until recently, I was only using Clojure for solving Project Euler problems (and for the sheer pleasure of learning it, of course). However, I decided that it was time to take it one step further. I wanted to see what it would be like to create a more realistic system in Clojure.

Background

For some time, I have been thinking about a pet project that has to do with organizing large amounts of PDF files. My wife and I have a lot of paperwork piled up around the house, and it seems like we will never find the time or energy to organize this mess. This is especially inconvenient for my wife. She is a music teacher, so she has an enormous amount of sheet music that is almost useless in its current form (because she can never find what she needs).

So what I want to do is simply buy a good scanner and scan all of it. Every piece of paper.

This will leave me with a lot of PDF files, some containing just a single page, some containing many unrelated pages. And so I need an archiving solution that supports at least the following features:

• Web-based, so I can reach it from anywhere
• Very easy upload, at least via web form, mail and network share
• Splitting and merging PDFs
• Annotating PDFs with metadata (name, description, tags etc)
• Very flexible search capabilities to find the stuff I need, fast

Although I’m sure someone has created this already (but please don’t tell me ), I thought it would make for an interesting spare time project. Also, I decided that Clojure should fit the bill nicely.

Technical considerations

Before I started coding, I actually spent a little time considering how to go about building this project:

• For the persistance layer, I wanted to use CouchDB, simply because I find it cool, and I want to learn more about it.
• For the UI, I’m still not sure, but I’ll probably settle on a JavaScript client and a REST backend written with Compojure. So no HTML generated on the server.
• PDF handling should be easy with some Java library (or ImageMagick if everything else fails).

Experiences so far

So I started coding and things went really well. Getting a PDF file uploaded and safely put into the DB was not too difficult.

Accessing CouchDB is easy using clojure-couchdb by Dan Larkin. The README file contains a lot of examples, and the library is generally very straightforward to use.

I was tripped by the fact that Compojure is designed to handle its own dependencies, including the Clojure jar itself! Of course, this is just a feature, not a requirement, but I managed to get a bit confused by it anyway. Also, I was unable to find clear, up-to-date documentation for the library, especially about file uploads.

Open issues

I really enjoy programming in Clojure, but DAMN, it is just so…different! In some regards, it’s like starting all over again, and I’m left with way more questions than answers.

Hopefully, some readers can help me in the comments:

• How can I do Continuous Integration (I’m using Git)? Never mind that, how can I even unit test my code? Has anyone tried using the approach of adding :test metadata directly on the functions to be tested?
• I tried stepping through my Clojure code with JSwat, but it didn’t work very smoothly. Is there a better alternative for debugging? Preferably from within Emacs.
• Is there a logging framework, or should I wrap and use one of the Java libs?
• It there a library for reading configuration files? I hope not, because this would be an excellent excuse for creating an internal DSL and just evaluate the config files.

I think it is very interesting to note here that CouchDB integration has been by far the easiest part of the project so far!

Maybe we need a “Practical Clojure” book in addition to Stuart Halloway’s excellent Programming Clojure?

/Jacob (chopmo.dk)

On my personal blog I’ve written part one of an article explaining PersistentHashMap. Those of you who attended my talk yesterday will recognize some of the diagrams.

The link is: http://blog.higher-order.net/?p=386

/Karl

I’ve upgraded all dcug members to the “author” role which means that you can post messages. Although this is the Danish Clojure Users’ Group, I suggest that we write in English, as it may be of interest to non-Danish speakers, who may/may-not live in Denmark.

I was talking to Jacob after my Aarhus Clojure talk yesterday, and he suggested I write a list of topics that users can write about – for inspiration. So here goes: Write about

* How you have been using, or plan on using, Clojure.

* Clojure vs. other LISPs

* Types of applications where Clojure shines.

* Types of application where Clojure doesn’t shine that brightly

* Articles explaining some technical aspect of Clojure

* Concurrency models, e.g., Actors (Erlang/Scala) vs Agents and Refs vs locks.

* Questions about how to solve some problem

* Clojure soluations to some problems.

* changing this list of topics; this list is immutable, really, so compute a new list from this one which has additional elements

I’ll post a more detailed description of how PersistentHashMap works soon, and also a Clojure solution to one of google’s “code jam” exercises…

/Karl

Upcoming events: I am giving a talk on Clojure in Copenhagen and Aarhus – this is the first chance for a dcug meetup (though also non-dcug members are invited). The events are after work and free – there will even be free sandwiches, compliments of Trifork

• Monday, Sept. 7th in Aarhus, location: Trifork. Registration: https://secure.trifork.com/aarhus-2009/freeevent/register.m?eventOID=2129
• Wednesday, Sept. 9th in Copenhagen, location: Trifork Cph. Registration: https://secure.trifork.com/aarhus-2009/freeevent/register.m?eventOID=2130.
• Free Clojure workshop at JAOO – featuring Rich Hickey. October 6, 2009, 17.30 – 19.30. Registration: https://secure.trifork.com/aarhus-2009/freeevent/register.m?eventOID=2093

Abstract for Aarhus/Cph talks.
Clojure is..

… a new functional, dynamic programming language for Java Virtual Machines. The primary novelty of Clojure is its strong focus on and support for in-process concurrency: a unique concurrency model, combining a notion of persistent (i.e., immutable, fast) data structures, with a lock-free concurrency model. This simplifies concurrent programming greatly and has good scalability properties.
Influenced by LISP and Haskell, Clojure supports pure, lazy functional programming and has a powerful macro system which makes extending the language to support DSLs easy and powerful.
This talk..
… is split in three parts. In the first part, Clojure is introduced for those who don’t know the language. There is so much to cover that this will be a fast tour with pointers to more information, but we will emphasize the unique aspects of the language.

In the second part we go into more depth regarding the implementation of persistent (and transient data structures) – “the secret sauce of Clojure”

In the third part we get to see Clojure in action running on some very cool technology – a unique opportunity! Azul Systems (www.azulsystems.com) has promised to make available one of their large Vega 3 compute appliances (864 core, 368 GB memory, let’s go concurrent). We will explore how the Clojure concurrency model fares in practice, scaling a demo of a parallel Traveling Sales Problem algorithm. We will also push the implementation to its limits in a high-contention demo. Great fun!

DCUG members can now get a 15% discount on JAOO tickets ordered before August 31st.

Simply click the banner below, choose “register here” and use the promotion code: dcug

I’ll get back soon with more Clojure news at JAOO…