Claquette - Animated Screenshots

Tuesday 02. June 2020 16:37:01 UTC+02:00 - Thomas Zoechling

This post was originally written as guest entry on the Eternal Storms Blog .

Animated GIF is not a good video format.
Actually it even isn't a proper video format because it lacks random access and audio support. Nonetheless animated GIFs experienced a rennaisance in recent years.
My theory is, that the format's success doesn't stem from the features it has, but from the ones it lacks:

• No random access: Defaults to autoplay – No playback interaction needed
• No sound: Guarantees silence – Always "Safe for Work"
• No support for 32bit colors: Moderate file sizes and a characteristic look

Given those constraints, GIFs are a great way to communicate simple concepts or interactions.
Want to showcase an animation or an application feature? Instead of a static screenshot with paragraphs of text, a 3 second GIF works equally well and also draws more attention.

In my daily life as software developer, I often need a quick way to capture graphical bugs. Those clips can be easily shared with colleagues or included with bug reports.
Sometimes it's also simpler to attach a GIF to customer support requests instead of explaining how a certain feature works.
To cover my own needs, I wrote a small macOS application that allows me to record the screen and export the result as animated GIF. The app uses a custom recording codec and also covers the rest of the GIF creation workflow like crop, trim and file size optimization.

You can download Claquette from the Mac App Store. For detailed product information, please visit our website .

Development

​When I started to implement the Animated GIF feature for Claquette, I began with a naïve approach.
Armed with several years of experience in writing Objective-C and some knowledge about video APIs on the Mac, I wrote a small prototype. That first version just read a video file frame by frame and sent the resulting frames to ImageIO. ImageIO is a system framework that supports reading and writing several file formats. It also comes with a basic Animated GIF encoder and so I decided to skip any third party libraries and just use the built-in mechanisms of macOS.
I was able to come up with a working prototype in a single afternoon. The program was just a simple command line utility, but it was able to turn an arbitrary video file into an animated GIF.

There was just one problem... Or actually there were several of them: Due to the inner workings of ImageIO, the program used vast amounts of memory. Also, the encoding took very long and the files it created were huge. On top of all that, the resulting GIFs looked horrible.
So while it only took me one Sunday afternoon to create a working prototype, it took me several months to fix the above issues. Especially the large file size and the bad visual appearance of the resulting GIFs required a lot of research.

Getting the most out of a 30 year old file format

The original GIF specification (GIF87a) was written in 1987 - almost 30 years ago. Animation features were added in GIF89a, which is still the most recent version of the format.
So how is it possible that a file format designed for low resolution screens and 256 colors is still in use today?
It turns out that the GIF specification contains some sections that open room for exploitation. Additionally the visual nature of GIFs allows for optimizations, that trick human color perception.

The format is simple and has the following basic structure:

1. Header
2. Logical Screen Descriptor
3. Global Color Table
4. Graphic Control Extension (Frame #1)
   • Delay
   • Disposal Method
   • Local Color Table
   • Image Descriptor
   • Image Data
5. Graphic Control Extensions (Frame #2)
6. Graphic Control Extension (Frame #3)
7. ... (1 GCE for each animation frame)
8. Trailer

Header and Trailer are basically just magic numbers that mark the start and the end of the file. The Logical Screen Descriptor contains some general image information like width, height and background color. The Global Color Table is a simple list of colors that may contain a maximum of 256 entries.
Main image information is stored in one ore more Graphic Control Extension blocks.

Color Table Generation

The color table sections of the format specification are a good starting point to optimize the visual quality of an animated GIF.
Both palettes are restricted by the format's infamous 256 color limit. When reducing an image that uses 32bit (16777216 colors, 256 alpha values) to 8bit (255 colors, 1 alpha bit) it becomes really important which colors you leave out. The process of reducing large palettes to small ones is called Color Quantization . Choosing a good quantization algorithm is crucial when aiming for visually similar images with a reduced palette.
Naïve quantization implementations are based on occurrence, where seldom used colors are left out in the final image. More elaborate algorithms use techniques like dimensional clustering or tree partitioning .

When developing for Apple devices there are several libraries that provide color quantization functionality. macOS and iOS even have basic color quantization algorithms built-in. Apple's implementation is part of the ImageIO framework's CGImageDestination API.

The following sample images were created using different quantization techniques. They illustrate the quality impact of the used algorithm on the final image.

The first image shows the output of CGImageDestination . The resulting colors are noticeably off. Apple's encoder also messes up the transparency color in the GIF palette, which leads to holes in some areas of the image (e.g. the titlebar).


The open source library FFmpeg also includes a color quantizer. FFmpeg produces way better results than CGImageDestination. The colors are more vibrant and the transparency color is set correctly.



The color quantization library used by Claquette also outputs a good choice of colors. Additionally the app performs color matching to avoid color shifts and to guarantee correct gamma values.

Frame Difference Calculation

Another important factor during GIF generation is efficient frame difference calculation.
The disposal mode in the Graphic Control Extension allows an encoder to specify how the context is set up before the next frame gets rendered.
GIF89a defines 4 disposal modes:

• Unspecified: Replaces the existing canvas with the full contents of the current frame.
• Do not Dispose: Leaves the existing canvas as-is and composites the current (sub)frame over it.
• Restore to Background: Sets a defined background color and draws the current frame. Areas outside of the subsection in the Image Descriptor shine through.
• Restore to Previous: Fully restores the canvas to the last frame that did not specify a disposal method.

The Image Descriptor section can be used to define a sub-image which does not provide pixel data for a full frame. Instead it contains coordinates and pixel data for a subsection of the full image. By using frame differences and sub-images with a proper disposal mode, redundant image data can be avoided. Depending on the nature of the input video, this can greatly reduce the file size of the final GIF.
Modern video codecs like H.264 use techniques like macro blocks and motion compensation. Those methods introduce small errors that propagate from frame to frame. Propagated errors show up as noise when calculating frame diffs and eventually lead to unnecessary large files.
Claquette uses a custom lossless codec, that only stores regions that actually changed. This guarantees exact frame diffs.

The following images show the difference between frame #1 and frame #2 of a screen recording. The only effective change between those frames is a change in the mouse cursor location. An exact diff should therefore only contain an offsetted cursor image.

The above diff image was created between 2 frames of an H.264 encoded movie. The visible noise is a result of intra-frame encoding errors.

The second image was created by Claquette's image diff algorithm. It only contains the mouse cursor - The only image element that actually changed between frame #1 and #2.

Finishing Touches

After implementing technical details like encoding and optimization, there were still some features missing. Claquette needed an editing UI to handle the whole GIF creation workflow.
As I wanted to keep the app lightweight and simple to use, I decided to add only a small set of editing capabilities: Trim and Crop.
Claquette uses AVFoundation, and therefore I was able to use the AVPlayer class, which comes with a built-in trim tool.
Crop functionality was a bit harder to implement. AVFoundation doesn't provide any UI component to display crop handles so I had to implement my own.
Besides the standard drag & move interface, my implementation also provides some unique features. It offers alignment guides with haptic feedback and the possibility to enter crop coordinates.

You can see the final implementation of the crop utility in the animated GIF below.

Launch

To launch the animated GIF feature, I prepared a press kit and wrote mails to review sites that mostly cover Mac software.
Additionally I submitted the app to Product Hunt and informed Apple's App Store Marketing team.
I can really recommend to launch on Product Hunt: Claquette received over 400 upvotes and finished in the top 3 on launch day. The site also hosts a friendly community, that makes extensive use of the Q&A section below each hunted product.
I was also lucky to get some mentions from high profile Twitter users and good App Store reviews.
Two weeks after launch, Apple suddenly moved Claquette into the "New and Noteworthy" feature section on the Mac App Store front page. Of course this also lead to a noticeable spike in the sales charts.
Overall I was very pleased with the release and the reception of Claquette 1.5.

Implementing Support for Continuity Camera

Thursday 18. October 2018 12:39:43 UTC+02:00 - Thomas Zoechling

One of macOS Mojave's user-facing features is Continuity Camera. It allows you to capture images or documents from your iOS devices.
While Apple prominently showcases the feature on the Mojave product page, documentation for developers is very thin. There's only a short mention of the related APIs in the "What's new in Cocoa" WWDC 2018 session.
Some of AppKit's default view classes support the feature out of the box (e.g. NSTextView ), but it's also possible to add capturing to your own controls.

Apple implemented Continuity Camera on top of the Services infrastructure of macOS and NSResponder . Captured image data is passed in via NSPasteboard .
To make the "Import from iPhone or iPad" menu item appear in a menu, we have to implement validRequestor(forSendType:returnType:) in a NSResponder subclass (e.g. in a view controller). From there, we need to return an instance that implements the NSServicesMenuRequestor protocol.

I created a small sample project that provides a simple implementation of the above methods. It consists of a view controller that hosts an NSImageView and a button. The button has an empty menu connected, which gets popped-up when the button's action is performed. The view controller is a subclass of NSResponder and it also implements NSServicesMenuRequestor . So when the macOS Services architecture traverses the responder chain, it asks if we accept images from the pasteboard via validRequestor(forSendType:returnType:) . As we indicate true by returning self , the menu gets populated with the Continuity Camera menu items. Those items dispatch their actions via the Responder Chain, so it is important that the view controller that handles the passed in image is firstResponder . Otherwise, menu validation fails and the Continuity Camera items are disabled. To ensure that, we call self.view.window?.makeFirstResponder(self) before showing the menu.

After the user captured an image on the selected iOS device, we need to handle the returned image in readSelection(from pasteboard: NSPasteboard) (which is part of NSServicesMenuRequestor ). The sample app reads the captured image from the pasteboard and displays it in an image view. You can download the Xcode project here .

Special thanks go out to Michael Schwarz . He implemented Continuity Camera support for MindNode for Mac and shared his insights with me.

Introducing ZIP Foundation

Tuesday 04. July 2017 16:28:30 UTC+02:00 - Thomas Zoechling

The Cocoa developer community already provides several useful and well-maintained ZIP libraries. But while working on my next app, I realized that I have some requirements that were not met by the existing libraries I found. So I set out to create my own with the following design goals in mind:

• High performance
• Random access to specific entries within an archive
• Deterministic memory consumption
• Stream based API to read from or write to archives
• Low-maintainance package dependencies (ideally the ZIP library should not pull in any 3rd party code at all)
• Usable on the server (Linux)

I recently open sourced my results on GitHub. The project is called ZIP Foundation and it is a Swift framework to create, read and update ZIP archive files. It uses Apple's libcompression to handle de- and encoding of compressed archives.

Unlike most other Cocoa ZIP framework authors, I decided against wrapping an existing C implementation (like the excellent minizip ) and implemented the ZIP specification by myself. By doing that, I learned a lot about the format, which later allowed me to write a minimal and modern implementation of the specification.
Writing everything from scratch also has the nice side effect that ZIP Foundation has no 3rd party dependencies and that it was relatively easy to port it to Linux.
Another design goal was a modern, Swift-style API. The API should provide easy to use high-level methods on one hand and fine grained low-level access to ZIP entries on the other.

The high level API is implemented as extension to FileManager and only consists of two methods: zipItem and unzipItem .
It allows you to zip and unzip files and directories with a single line of code:
try fileManager.zipItem(at: sourceURL, to: destinationURL)

and
try fileManager.unzipItem(at: sourceURL, to: destinationURL)

To access individual entries, you can use the Archive class. Archive conforms to Sequence and therefore supports subscripting. To access an arbitrary entry within an archive you can simply write:
guard let entry = archive["file.txt"] else { return }

Archive also provides a closure-based API to access and modify ZIP files. This allows you to perform ZIP operations on-the-fly, without round trips to the file system.
To learn more about that, please refer to the README on GitHub or the method documentation in Xcode.

Performance

High performance was very high on my requirements list and so I designed the framework for it right from the beginning.
With macOS 10.11/iOS 9.0, Apple introduced a new compression library: libcompression. It comes with several compression encoders and decoders, including a highly optimized version of the zlib decoding algorithm. Zlib is available on virtually all platforms and its deflate implementation is the most common compression method used for ZIP archives. Besides its wide availability, zlib also provides a good balance between compression ratio and encoding/decoding speed. It is open source and has been constantly improved since its initial release in 1995. Apple somehow managed to speed-up the decoding part of zlib by 60% while still maintaining compatibility. Given that impressive decoding speed, using libcompression for ZIP Foundation was an obvious choice. The library is used on all Apple platforms (iOS 9.0+ / macOS 10.11+ / tvOS 9.0+ / watchOS 2.0+). On Linux, ZIP Foundation falls back to zlib.

For the following performance tests, I wrote a small command line utility that performs extract, add and remove operations with various entry sizes. The baseline performance is established by the default zip/unzip command line tools on macOS. Each operation is performed 10 times for each test entry size. The final result is the average of the ten runs in milliseconds. ZIP Foundation was compiled with -Ofast and -whole-module-optimization . The benchmark was executed on macOS Sierra 10.12.5. You can download the utility here .

During extraction, ZIP Foundation heavily benefits from the fast zlib decoding in libcompression. When taking into account how common zlib decoding is in everyday computing (not only for ZIP files but also for ubiquitous formats like PNG), it soon becomes clear that Apple's engineers really helped us to save a lot of time and battery with that optimization.

In WWDC Session 712/2015 , the libcompression engineers mention that they only optimized the decoding part of zlib. The graphs for addition and removal clearly show that. The small gains were all achieved by tweaking IO and compression buffer sizes.

Code Quality

As ZIP is often used as a container format for proprietary file formats, reliability and performance should be a high priority for every ZIP framework. ZIP Foundation is my first non-trivial project that is fully covered by unit and performance tests.
I don't think that achieving 100% coverage will instantly guarantee a certain level of quality. However, I found that while aiming for it, I had to permanently question program flow, API design and type choices. That lead to a lot of refactoring which in the end made the framework more robust and easier to use. Another advantage of 100% coverage is, that it uncovers dead code. If certain code paths are not exercised after refactoring, they are either dead or the refactoring has introduced flawed logic. Both aspects are also noticeable in code bases with less than 100% coverage, but the root causes would not be as easy to spot. (To easily find uncovered code, it helps to have Code Coverage Visualization turned on .)

Non-Apple Platform Support

When Apple open sourced Swift, they also made an initial push to bring the language to non-Apple platforms. As ZIP Foundation does not use any platform specific dependencies, I decided to support Linux right from the beginning. Apple provides a Swift implementation of Foundation , so it was possible to get a buildable framework with very little effort. I used Ole Begemann's excellent tips to get a Linux test environment for Swift projects using Docker. While getting the project into a compilable state was not too difficult, I soon ran into a wall when trying to execute the project's test suite. The main issues all stemmed from the fact that Apple's open source implementation of Foundation.framework is far from feature complete at the moment (The current Swift development version at the time of writing is 4.0). For instance fundamental methods like FileManager replaceItem are missing. Another class that currently only offers a subset of the real Foundation.framework was (NS)URL, where things like retrieving file metadata via resourceValues(forKeys keys: [URLResourceKey]) is not possible. Luckily Swift has built-in interoperability with C APIs and so I was able to use POSIX counterparts of the aforementioned methods to work around most methods that threw NSUnimplemented .

Conclusion

ZIP Foundation was my first 100%-Swift project and I have learned a lot about ZIP, Swift and framework development along the road.
I used Swift since its introduction, but mostly for high level UI code and app development. During the course of this project, it also turned out as good fit for lower level tasks such as large file processing and compression.