A journey from Elm 0.18 to 0.19
The good, the bad, and the ugly
On November, I landed a new job at Prima Assicurazioni as a full stack developer, and I was put on a fairly medium-sized project: a webapp written in Elixir, GraphQL and Elm.
In the early days I was trying to get confident with the languages as well as the codebase — functional programming was something kind of new to me. While learning all these new, exciting stuff, I noticed a strange PR:
I imagined it being something in the likes of a webpack 3-to-4 migration, so I asked a coworker about it and I was left with this:
That’s a spectacularly failed attempt at migrating to Elm 0.19… There are so many breaking changes between 0.18 and 0.19 that we wasted three weeks trying to migrate, and didn’t make anything out of it, except for frustration and rage.
I didn’t know how to take this revelation. Of course it seemed like a cool challenge, but after learning something more about Elm, I started to think that even trying to migrate would be something for fools.
A bit of history
Elm started in 2012 as a pet project of Evan Czaplicki, with the aim of developing a language for the frontend without all those nasty runtime exceptions typical of Javascript, along with a strict type system. He wrote the compiler in Haskell, because he wanted to recreate the same great experience, namely: if it compiles, then it works.
The other big selling point of Elm is that it’s very opinionated about how you write your frontend code. No DOM manipulation, no jQuery-like syntax, no convoluted architecture. Components are instead written in a declarative fashion, and a virtual DOM implementation deals with the actual render. A View-Update-Model architecture is employed for state management: messages (triggered from various actions in the view, or from requests, sockets, etc) are intercepted in Update functions, which update the Model, and again may change the declarative View.
If it sounds familiar, it’s because I’ve just described the React+Redux architecture. Here is the talk where Dan Abramov first introduced Redux, which democratized state management in modern frontend development. This process was largely inspired by Elm.
Elm 0.19
The advantages offered by Elm seem great and obvious — and they actually are. However, the language is still mostly a one-man effort, and it’s still in version 0.x after seven years of development.
After almost two years from the previous version, on late August 2018 Elm 0.19 came out with many incredible new features: tree shaking, compilation time reduced by a factor of 100, a completely new standard library, and much much more.
But all that didn’t come for free. Most of the included libraries were refactored and included in the new elm namespace, while elm-lang was deprecated. The package system was redesigned, effectively making it impossible to install packages from other sources than the official repository. You can’t fork a library and specify it as a dependency: either you publish it, or hope the maintainer likes your PR.
The biggest incompatible change was the removal of the possibility to implement so-called effect managers, which is now a feature only for packages in the elm namespace. The official motivation is that it was something undocumented, never intended for package authors or developers to implement their own, thus it was banned altogether. It happened out of the blue, without a migration plan, and many people didn’t react well.
Finally, not all the existing packages were ported to the new version. The websocket module of the standard lib has been deprecated and there is still no clear plan for it.
The migration
So, without further ado, let’s dive into the actual migration. How to accomplish it correctly, efficiently, and without breaking too much? These are the steps you should follow:
- Use automatic migration tools, if available, to get a global overview of the affected parts of your application and automate boring stuff.
- Prepare a detailed, independent plan for every deprecated functionality or library.
- Create isolate modules to interact with external dependencies: such a refactor will be insanely useful for the current migration, and will simplify future ones as well.
- Develop new features in a backward compatible way: if the migrated module is both compatible with the old and the new version of the language, you enter a healthy feedback loop which allows you to iterate quickly, and see the light at end of the tunnel.
- Don’t rush. Seriously, don’t do it. Don’t let the excitement for the upgrade take over you and make you forget everything else.
With that in mind, we collected some ideas and started working on a plan to make the migration as smooth as possible. All the steps described below include a comparison between how we estimated the tasks in terms of time and complexity, and how they actually turned out to be.
Step 1: automatic migration tools
Estimate: easy. Reality: easy
We used the wonderful elm-upgrade tool to have a glimpse of how much stuff would be broken after transitioning to Elm 0.19.
By just running the command elm-upgrade in the root folder of our project, the elm-package.json file was analyzed and we were asked which libraries we wanted to update, remove or keep. At the end of the process, the elm.json file was created with all the new dependencies, along with a log that listed all the other libraries that instead failed the update, due to incompatibility issues with Elm 0.19.
Then, it proceeded to refactor all our application code by replacing deprecated syntax (such as the exclamation mark ! for tuples), old module imports, and so on. In practice, it tried to automate all the changes that a couple sed commands would have resolved.
This process took a couple minutes, and left us with a diff that touched hundreds of files. It would have been a huge mistake to just commit and keep going with the migration. As I said above, it needs to be an incremental process, and the compatibility-breaking refactor made by elm-upgrade was one of the last steps that needed to be done.
This is the upgrade-warnings.txt file produced by the first run of elm-upgrade, listing all the libraries that were not ported to Elm 0.19, along with all the deprecated functionalities we were using from the standard library:
WARNING! 4 of your dependencies have not yet been upgraded to
support Elm 0.19.
- https://github.com/justinmimbs/elm-date-extra
- https://github.com/mgold/elm-date-format
- https://github.com/saschatimme/elm-phoenix
- https://github.com/simonh1000/file-reader
Here are some common upgrade steps that you will need to do manually:
- NoRedInk/elm-json-decode-pipeline
- [ ] Changes uses of Json.Decode.Pipeline.decode to Json.Decode.succeed
- elm/core
- [ ] Replace uses of toString with String.fromInt, String.fromFloat, or Debug.toString as appropriate
- elm/time
- [ ] Read the new documentation here: https://package.elm-lang.org/packages/elm/time/latest/
- [ ] Replace uses of Date and Time with Time.Posix
- elm/html
- [ ] If you used Html.program*, install elm/browser and switch to Browser.element or Browser.document
- [ ] If you used Html.beginnerProgram, install elm/browser and switch Browser.sandbox
- elm/browser
- [ ] Change code using Navigation.program* to use Browser.application
- [ ] Use the Browser.Key passed to your init function in any calls to Browser.Navigation.pushUrl/replaceUrl/back/forward
- elm/url
- [ ] Changes uses of Navigation.Location to Url.Url
- [ ] Change code using UrlParser.* to use Url.Parser.*
- elm/random
- [ ] Change references to Random.Pcg.* to Random.*The log speaks for itself, and the top section clearly suggests that most of the work will need to be done on three areas:
- How we deal with files
- The communication to the backend through phoenix sockets
- Date and times
In fact, most of the incompatibilities described in the latter section are either trivial, or easy but a bit time consuming. With that in mind, we decided to start dealing with something that would be compatible with both versions of Elm: the removal of native modules.
Step 2: native modules
Estimate: easy. Reality: medium
As Evan stated in the post I already linked above, native modules and effect managers are prohibited in Elm 0.19, and need to be replaced with pure Elm versions or Javascript ports.
We already removed a couple native modules before starting the migration, mostly for UUID generation and similar stuff, and it was an operation that took minutes at most.
Refactoring the code that was using the Elm file-reader library took a bit more time, as we were not that familiar with the FileReader browser API. Turns out that it’s really simple, and we only had to do a Javascript port that subscribed to the FileSelected Elm message, and sent in response the base64-encoded contents of the file. You can see a working example here.
While developing these features, the cool thing was that we could immediately test them on the codebase using Elm 0.18, being confident that it would have worked on Elm 0.19 too!
Then it came the time to think about the removal of elm-phoenix, the library that wraps Phoenix sockets and allows us to communicate with the Elixir backend. We employed websockets for two different things: querying the GraphQL server using Absinthe, and sending push notifications.
It turned out that elm-phoenix implemented an effect manager itself, and furthermore it used the deprecated websocket library underneath. We couldn’t use none of this libraries, so we had to look for some solid alternatives.
There is a cool websocket client for Elm 0.19, or we could always resort to Javascript ports. However the main problem wasn’t about the implementation, but about the scale of the refactor: we had hundreds of queries, and rewriting them without an effect manager would have taken days, possibly weeks.
At this point we were clueless and discouraged from the possibility of this huge refactor, until a coworker had a fantastic idea: change the transport.
Step 3: websockets
Estimate: impossible. Reality: easy
Websockets are cool and fast, but do you really need them for GraphQL queries? Probably not.
The fact that Elm 0.19 bundles the http library as an effect manager, helped a lot towards the decision of using HTTP as a transport for our queries. Also, it meant that we just had to refactor the one, main function responsible for making queries and decoding the response and/or error messages. Changing the backend took minutes as well, thanks to the huge flexibility of Absinthe.
We had just done in a couple hours what seemed to be taking weeks of work. That was the real game changer for us.
At this point we only had to reimplement socket initialization and error handling, which in our case just meant wrapping Phoenix sockets, since they already implement all these features internally.
Handling push notifications was as easy as we initially thought. We had only about a dozen different messages, but it would have been as easy even with an order of magnitude more. It’s all about making a couple of Javascript ports, declaring all the messages we are interested in handling, and have Elm call the appropriate actions.
That’s all for native modules. I have to admit that this stuff resulted in a week with more bugs than usual, mostly due to websockets. The Phoenix sockets library abstracted away all the details, and it turned out we didn’t have a thorough knowledge of websockets, especially about error handling, reconnections, auth issues, and so on. But bug fixes came, and we only had one last step before migrating.
At this point, we were completely sure that only date and time stuff needed to be ported, and it needed to be done in a totally different, non-Elm 0.18 compatible way (as explained below). Thus, we run elm-upgrade on the project and continued working from there.
Step 4: date and time
Estimate: easy. Reality: hard
Elm 0.19 completely changed the date/time handling, by introducing two new types, Posix and Zone, for representing respectively time in UTC format and timezones. The big difference with Elm 0.18 is that we now have a single type to deal with dates and times, rather than two separate ones that often clash together.
The library is very frugal and makes simple operations like parsing a date incredibly hard. In fact, we had to resort to an external library just for the purpose of parsing dates in ISO8601 format. That’s because you only get the Time.millisToPosix function, which basically accepts a UTC Unix timestamp and transforms it into a Posix. You’ve got a date as a string in ISO8601 format? You better write a parser yourself.
The Zone module does not allow you to specify timezones when parsing or formatting dates: you only get utc, or you can compute the local timezone using the here task. And that’s all. Do you want to make an app that displays the current time in three different timezones? You have two choices: either you are a package author and implement your own custom timezone or you use a 20KB external library that does it for you.
In a part of the application we had a couple <input type="datetime-local">, which returns selected dates as string using this format: YYYY-MM-DD HH:mm. We knew in advance they were computed using italian timezone, and we had to parse them, make some computations, and have the output again in italian timezone. There is literally no easy way to do that. We had to do a Javascript port that calls new Date(input).toISOString(), which returns a string in ISO8601 along with timezone, and can be fed to the external library that parses it into a Posix.
After way more work than we initially estimated, we ended up with a custom Date module that implements all the parsers and formatters that we use throughout the application. It really helped cleaning the code from calls to various external dependencies, but we still had to use three or four different libraries inside the central module, due to lack of basic features in the elm/time package.
Final steps
Estimate: easy. Reality: easy
The migration was 99.9% done. All we needed to do was refactor the navigation, as explained in the official docs, and use the new APIs in a couple more places, like application startup, regex parsing, and so on.
Finally, we could send the application to the QA team for testing, and wait for them to tell us that we broke pretty much everything :-)
Jokes aside, the migration took some weeks to complete. We broke a lot of stuff. But the most important thing was that we proceeded iteratively, from the start to almost the end (date and times modules excluded), so that no one in the team was focusing on Elm 0.19 alone. Everyone would in fact work on features that would be useful for both versions of the app.
So, was it worth it at the end? I’ll leave the numbers speak for me.
Benchmarks
The benchmarks were executed on a machine with an Intel i7–8550U CPU and 16GB RAM. The Elm webapp was in the range of 50 kLOC.
Commands were run by preceding the time command, and only real value is reported. Full benchmarks here and here.
Initial build times
The commands were executed after downloading elm dependencies locally, and deleting the elm-stuff folder before each run, so the cache is empty.
Incremental build times
The commands were run after doing a clean build, then making the described changes and finally build again.
📝 Read this story later in Journal.
🗞 Wake up every Sunday morning to the week’s most noteworthy Tech stories, opinions, and news waiting in your inbox: Get the noteworthy newsletter >