Vedang Manerikar

My corner on the Internet.

In/Clojure 2024: Developer Tooling For Speed And Productivity In 2024!

These are slides from my talk at In/Clojure 2024. These slides may not have all the context, they'll make better sense after you watch the talk.

If you have any problems using the tools I've mentioned here, or if you want help improving your Clojure programming workflow, drop me an email! (Or better still, book a session with me). I love talking to other Clojure programmers, exchanging tips and tricks, and learning from them.

Why this talk?

So you'll show me your cool workflows! (help me get better!)

I want to write a new library or application!

I need to figure out …

  • How to run my application
  • How to run all the tests in my project
  • How to build the artifacts of my app/lib
  • How to deploy these artifacts so others can use them

Speaker Notes

The Javascript ecosystem, for all the abuse we hurl at them, has done an incredible job. The templating that people have access to out of the box is something we should all aspire to!

I use deps-new!

https://github.com/seancorfield/deps-new

clojure -Tnew lib :name me.vedang/depsnew_lib

tree depsnew_lib

RESULTS:

depsnew_lib
├── CHANGELOG.md
├── LICENSE
├── README.md
├── build.clj
├── deps.edn
├── doc
│   └── intro.md
├── src
│   └── me
│       └── vedang
│           ├── depsnew_lib.clj
└── test
    └── me
        └── vedang
            └── depsnew_lib_test.clj

What deps-new gives me

  • All the boilerplate!
  • Clear Instructions in the README
  • Aliases:
    • clojure -X:run-x, clojure -X:run-m Run your app
    • clojure -T:build test Run tests
    • clojure -T:build ci Build artifacts
    • clojure -T:build deploy Deploy artifacts

You can even write your own deps-new templates!

AND YOU SHOULD!

(looking at you, framework authors!)

What can we as clojure devs do?

  • Does your favourite framework have a deps-new template? Add it!
  • Do you find yourself copying the same code everywhere? Try and extract it into a template!

References to learn deps-new

I want to add more aliases to my repository!

  • How do I connect to a REPL?
  • How do I setup logging?
  • …

I use neil!

https://github.com/babashka/neil

  • neil can add aliases for you!
  • neil can do so much more!

Examples of using neil!

neil add cider

:cider ;; added by neil
{:extra-deps {cider/cider-nrepl {:mvn/version "0.47.0"}
              djblue/portal {:mvn/version "0.52.2"}
              mx.cider/tools.deps.enrich-classpath {:mvn/version "1.19.0"}
              nrepl/nrepl {:mvn/version "1.1.1"}
              refactor-nrepl/refactor-nrepl {:mvn/version "3.10.0"}}
 :main-opts  ["-m" "nrepl.cmdline"
              "--middleware" "[cider.nrepl/cider-middleware,refactor-nrepl.middleware/wrap-refactor,portal.nrepl/wrap-portal]"]}

Neil can do more than just add aliases!

neil dep search next.jdbc

RESULTS:

:lib pro.juxt.clojars-mirrors.com.github.seancorfield/next.jdbc :version "1.2.674" :description "pro.juxt.clojars-mirrors.com.github.seancorfield/next.jdbc on Maven"
:lib net.cloudopt.next/cloudopt-next-jdbc :version "3.1.3.0-RELEASE" :description "net.cloudopt.next/cloudopt-next-jdbc on Maven"
:lib com.github.seancorfield/next.jdbc :version "1.3.999-SNAPSHOT" :description "The next generation of clojure.java.jdbc: a new low-level Clojure wrapper for JDBC-based access to databases."
:lib seancorfield/next.jdbc :version "1.2.659" :description "The next generation of clojure.java.jdbc: a new low-level Clojure wrapper for JDBC-based access to databases."
:lib com.layerware/hugsql-adapter-next-jdbc :version "0.5.3" :description "next.jdbc hugsql adapter"
:lib dev.weavejester/ragtime.next-jdbc :version "0.9.4" :description "Ragtime migrations for next.jdbc"
:lib hugsql-next-jdbc/hugsql-next-jdbc :version "0.1.3" :description "next.jdbc adapter for HugSQL"
:lib de.active-group/active-jdbc :version "0.2.1" :description "Functions to work with JDBC building upon next.jdbc."
:lib daaku/pgmig :version "2.0.1" :description "Database migrations for PostgreSQL using next.jdbc."
:lib taskmaster/taskmaster :version "0.0.4-SNAPSHOT-1" :description "Background publisher/consumer on top of Postgres, next.jdbc and hikari-cp"

Neil can do more than just add aliases!

neil dep add :lib com.github.seancorfield/next.jdbc

neil dep add :lib nextjournal/clerk :alias :cider

What can we as clojure devs do?

  • Make it easy to add aliases to neil!
  • I would like to add to existing aliases instead of creating new ones!

References for learning more about neil

Okay, I am now ready to spin up a REPL and connect to it!

clojure -M:test:logs-dev:cider

Speaker Notes

Wait, what's the logging alias?

Logging is hard!

So once I figured it out, I made a template!

I'm happy I never have to figure this out again!

neil add logs-dev

:logs-dev ;; added by neil
 {:extra-deps {me.vedang/logger {:local/root "logger"}}
  :jvm-opts
   ["-Dclojure.tools.logging.factory=clojure.tools.logging.impl/log4j2-factory"
    "-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.Slf4jLog"
    "-Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector"
    "-Dlog4j2.configurationFile=logger/log4j2-dev.xml"
    ;; Change logging.level to one of TRACE, DEBUG, INFO, WARN, ERROR
    ;; depending on requirement during development
    "-Dlogging.level=DEBUG"]}

Templates for the win

clojure -Sdeps '{:deps {io.github.vedang/clj-logging {:git/sha "e009d366c827705f513ef9018ffd920a49ce19da"}}}' -Tnew create :template me.vedang/logger :name me.vedang/logger

tree logger

RESULTS:

logger
├── README.org
├── deps.edn
├── project.clj
├── resources
│   └── logger
│       ├── log4j2-dev.xml
│       └── log4j2-prod.xml
└── src
    └── me
        └── vedang
            └── logger
                └── interface.clj

7 directories, 6 files

I use structured logging everywhere!

http://pedestal.io/pedestal/0.7/reference/logging.html

(in-ns 'me.vedang.depsnew-lib)
(require '[me.vedang.logger.interface :as log])
(log/info :function :welcome
          :action :starting
          :args {:greeting "Hello World"})

{
  "instant": {
    "epochSecond": 1711161317,
    "nanoOfSecond": 787564000
  },
  "thread": "nREPL-session-d6c4a5db-e57c-4efe-906b-7e5f1d0ad6e6",
  "level": "INFO",
  "loggerName": "me.vedang.depsnew-lib",
  "message": "{\"function\":\"welcome\",\"action\":\"starting\",\"args\":{\"greeting\":\"Hello World\"},\"line\":3}",
  "endOfBatch": true,
  "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
  "contextMap": {},
  "threadId": 41,
  "threadPriority": 5
}

Prefer emitting logs as JSON events. This lets us push them to Observability tools easily in production.

I want beautiful Documentation

Clojure is REPL-first! Where are my notebooks?

Speaker Notes

We've been writing notebooks and refactoring them into usable code-bases all our lives. So why do we not have beautiful looking notebooks like the Python people?

I use Clerk: write beautiful notebooks and beautiful code-bases!

https://github.com/nextjournal/clerk

Write your comments like you write Markdown, done!

Clerk: write beautiful notebooks and beautiful code-bases!

;;; # 📈 Sunday Morning Fun With Our Group Meditation Data

;; First, we need to parse the data from the Whatsapp Group. Whatsapp
;; provides a `export-chat` feature that we can use, but there are
;; some things we need to fix first:

;; ##  Whatsapp uses a ridiculous format for the date when the message was sent
^{:nextjournal.clerk/visibility {:code :show :result :hide}}
(def whatsapp-export-date-format
  "Frankly, this is a ridiculous format that no one should use."
  "MM/dd/yy, hh:mm a")

;; The problem isn't so much the format, as the fact that they make it
;; hard to parse. For example, they won't export the date as
;; `"05/01/23, 03:31 pm"`. They will export it as `"5/1/23, 3:31 PM"`.
;; Every part of this is meant to fail your parser. 🤷🏽‍♂️

What Clerk notebooks look like:

Speaker Notes

  • Talk about Separedit!

I use tagref: clear separation of code and documentation

https://github.com/stepchowfun/tagref

(require 'clojure.math)

;; [tag:polynomial_nonzero] This function never returns zero.
(defn polynomial [x]
  (+ (clojure.math/pow x 2) 1))

;;; in some other file
(defn inverse-polynomial [x]
  ;; This is safe due to [ref:polynomial_nonzero].
  (/ 1 (polynomial x)))

Documentation for the team: tagref

tagref list-tags | awk '{print $1}'

RESULTS:

[tag:gifs_need_to_be_converted_to_video_format]
[tag:rules_for_deciding_when_we_should_update_a_concept_record]
[tag:an_important_thing_to_understand_about_repetition_records]
[tag:rules_for_deciding_when_we_should_update_a_chapter_record]
[tag:rules_to_mark_a_chapter_as_done_for_now]
[tag:edge_case_empty_concepts_when_updating_chapter_or_concept_records]
[tag:the_order_in_which_parsers/checkers_run_is_important]
[tag:next-question_is_the_heart_of_our_logic]
[tag:rules_for_picking_the_next_concept_for_this_student]
[tag:rules_for_picking_the_next_chapter_for_this_student]
[tag:field_name_change_needs_to_be_the_last_parser]
[tag:no_need_to_optimise_insertion_db_calls]
[tag:clojure.string/replace_using_replacement_function]
[tag:what_can_session_filters_contain?]
[tag:rules_to_mark_a_concept_as_done_for_now]
[tag:extend_protocols_for_automatic_conversion_of_data_to_and_from_pg]
[tag:meaningful_ring_responses]
[tag:what_to_keep_in_mind_when_updating_routes]
[tag:on_magic_links]

References for improving documentation

I want to write clean, maintainable code!

https://github.com/clj-kondo/clj-kondo https://github.com/weavejester/cljfmt https://github.com/kkinnear/zprint

I do automatic Linting and Formatting on the code base!

clj-kondo: My recommendation

Export clj-kondo config for libraries you use! (create .clj-kondo directory at root first)

clj-kondo --lint "$(clojure -A:dev:test:cider:build -Spath)" --copy-configs --skip-lint

(clojure-lsp will do this automatically)

tree .clj-kondo

RESULTS:

.clj-kondo
├── babashka
│   └── fs
│       └── config.edn
├── http-kit
│   └── http-kit
│       ├── config.edn
│       └── httpkit
│           └── with_channel.clj
├── nextjournal
│   └── clerk
│       ├── config.edn
│       └── nextjournal
│           └── clerk
│               └── viewer.clj_kondo
└── rewrite-clj
    └── rewrite-clj
        └── config.edn

12 directories, 6 files

clj-kondo: My recommendation

Commit your .clj-kondo folder!

cljfmt: My recommendation

(For Emacsen) Use Apheleia!

Runs code-formatter on the buffer "at the right time" with minimal disturbance.

What can we as clojure devs do?

Add clj-kondo configuration for your favourite libraries!

References for clj-kondo, cljfmt and zprint

Can we improve the REPL experience?

After all, this is where we are all day!

Speaker Notes

IF we can make the experience even 1% better, isn't that something amazing?

Debugging like I've never experienced before: Flowstorm!

neil add cider-storm

:cider-storm ;; added by neil
 {:classpath-overrides
  ;; we need to disable the official compiler and use ClojureStorm
  {org.clojure/clojure nil}
  :extra-deps {cider/cider-nrepl {:mvn/version "0.47.0"}
               com.github.flow-storm/clojure {:mvn/version "1.11.2"}
               com.github.flow-storm/flow-storm-dbg {:mvn/version "3.13.1"}
               djblue/portal {:mvn/version "0.52.2"}
               mx.cider/tools.deps.enrich-classpath {:mvn/version "1.19.0"}
               nrepl/nrepl {:mvn/version "1.1.1"}
               org.openjfx/javafx-controls {:mvn/version "23-ea+3"}
               org.openjfx/javafx-base {:mvn/version "23-ea+3"}
               org.openjfx/javafx-graphics {:mvn/version "23-ea+3"}
               org.openjfx/javafx-swing {:mvn/version "23-ea+3"}
               refactor-nrepl/refactor-nrepl {:mvn/version "3.10.0"}}
  :main-opts  ["-m" "nrepl.cmdline"
               "--middleware" "[flow-storm.nrepl.middleware/wrap-flow-storm,cider.nrepl/cider-middleware,refactor-nrepl.middleware/wrap-refactor,portal.nrepl/wrap-portal]"]
  :jvm-opts ["-Dclojure.storm.instrumentEnable=true"
             "-Dclojure.storm.instrumentOnlyPrefixes=me.vedang."]}

Debugging with Flowstorm

(in-ns 'me.vedang.depsnew-lib)

(defn foo [n]
  (->> (range n)
       (filter odd?)
       (partition-all 2)
       (map second)
       (drop 10)
       (reduce +)))

References for learning more about Flowstorm

Why Micro-services?

Smaller teams, clear ownership!

Why Micro-services?

  • Easier to test
  • Clear contracts (API)
  • Independent deployments

Why Monorepos?

  • No version hell
  • Easy refactoring across multiple services (in theory)
  • Easy collaboration and on-boarding (in theory)

So what goes wrong with Monorepos?

  • Contracts are unwritten
  • Testing becomes messy (specifically, test-suite time)
  • Refactoring, Collaboration, On-boarding Hell

Speaker Notes

Eventually, teams start building conventions and tooling and start refactoring.

The problem is, it's too late by now.

I use Polylith from day one

https://github.com/polyfy/polylith

Polylith is a set of conventions and tooling that has thought through all these problems!

Let's create our Polylith monorepo!

poly create workspace name:polylith top-ns:me.vedang :commit

tree -L 3

RESULTS:

.
├── bases
├── components
├── deps.edn
├── development
│   └── src
├── logo.png
├── projects
├── readme.md
└── workspace.edn

6 directories, 4 files

Speaker Notes

Addendum: you can also create a workspace in an existing repo: poly create workspace top-ns:me.vedang

The terminology of Polylith: project

projects/ : Interface with hosting platform

Let's think through an example:

  1. Static Site : projects/weblogstatic (Artifact: Public Folder, Deployment: Github Pages)
  2. Dynamic Site : projects/weblogdynamic (Artifact: Uberjar, Deployment: Dockerfile, fly.toml)

The terminology of Polylith: base

bases/ : Interface with the external world

Continuing with our example:

  1. Static Site: bases/serverstatic
  2. Dynamic Site: bases/serverdynamic (server! routes)

Speaker Notes

This folder contains code that interacts with the external world

Think: CLI tool, Server, Lambda function, Worker

The terminology of Polylith: component

components/ : Interface with a module of business logic.

Continuing with our example:

  1. components/content: The markdown files I want to serve as posts
  2. components/render : Code to convert MD files to HTML
  3. components/readers: Logged-in visitors on my dynamic website

Speaker Notes

This folder contains re-usable, modular code that implements specific functionality.

How does all this tie together?

projects/weblog_static -> [bases/server_static
                           components/content
                           components/render]

projects/weblog_dynamic -> [bases/server_dynamic
                            components/content
                            components/render
                            components/readers]

How Polylith promotes Modular architecture

Code outside a component can only refer to functions in the component's interface namespace.

Speaker Notes

During development, you have access to the entire code-base! But there are rules around how you can access any code outside your own brick.

(so I don't need to know how you implement the code, I'm only concerned with the interface of your component)

Interface functions pass-through the arguments to appropriate internal functions.

Polylith: getting the best of micro-services and mono-repos

  • Where does the Dockerfile go? projects/weblog_dynamic
  • Where do I put end-to-end tests? projects/weblog_dynamic/test
  • Where do I define the routes and the main class? bases/server_dynamic/core.clj
  • Which functions do I use to render HTML pages? components/render/interface.clj
  • Which functions do I write contract tests for? components/render/interface.clj
  • Which functions do I document thoroughly? components/render/interface.clj
  • Where do I hook observability? components/render/interface.clj, bases/server_dynamic/core.clj
  • Where does the actual rendering functionality live? render/posts.clj, render/tags.clj, render/index.clj …

What does the poly tool give me?

  • Enables running the right tests
  • Checks to ensure component code is used properly everywhere
  • Upgrades libraries across all your bricks

… and much more

References for learning more about Polylith

There is so much more …

  • CI/CD! (Use babashka!)
  • Observability (pedestal! or iapetos + clj-otel)
  • Web Development (martian makes talking to APIs a breeze!)

Come say hi!

  • @vedang on 🐘fosstodon, 🖥️github, 🐦twitter
  • https://unravel.tech
    • We can help with your problems!
    • (design, product, engineering)

THANK YOU! QUESTIONS?

Published On: Wed, 27 Mar 2024. Last Updated On: Sun, 15 Sept 2024.

Uplinks

Jumps