Fixed-Dimensions Images

In order to frame this use case, think of a layout that resembles Figures 11-1 and 11-2.

Figure 11-1. Sample fixed-dimensions image in a responsive layout—wide viewport

Figure 11-2. Sample fixed-dimensions image in a responsive layout—narrow viewport

As you may have noticed, the image in these layouts remains in the same dimensions regardless of the layout changes that result from the responsive design, the same as images in nonresponsive designs. So why would we consider this image “responsive”?

Well, the problem starts when you’re browsing that site over a high-resolution screen, and notice significant blur.

Retina screens “need” images that are twice (or more) the resolution of traditional resolution screens. If the image we provide the browser in that case is of lower dimensions, the browser will upscale the image, resulting in blurriness.

So, how do we resolve it? The first reaction from web developers was to upgrade their images, and serve larger images to their entire audience. The problem with this was that they were now serving larger images to all devices, including the ones that have absolutely no need for images that are twice as large.

For these devices, the result of the larger images was, besides the bandwidth costs and delay, increased CPU costs and higher memory consumption, as larger images had to be decoded and then stored in memory. See Chapter 9 for more details.

Variable-Dimensions Images

The variable-dimensions image use case is slightly different from the previous one, since it’s tightly related to responsive websites. Consider the layouts shown in Figures 11-3 and 11-4.

Figure 11-3. Sample variable-dimensions images in a responsive layout—wide viewport of a desktop breakpoint

Figure 11-4. Sample variable-dimensions images in a responsive layout—narrower viewport of a desktop breakpoint

In this case, larger viewports need larger images; otherwise, the images will be blurry. But, similiarly to the previous use case, higher-resolution screens also need larger images. Again, the initial developer response was to send the largest possible image, but that’s hardly scalable. In a world with 28-inch high-resolution screens, the largest possible image can be pretty big. That’s not something that you want to send down a mobile connection to your average user. That case is sometimes referred to as “download and shrink”: you’re downloading bytes that users don’t necessarily need, burdening their mobile data plan and slowing down their experience.

Art Direction

What happens when your images are materially different in the various responsive breakpoints? When you want to adapt the images to the different breakpoints so that they will be clearer in the context of the different layout?

Well, that’s a different use case from what we’ve seen before. A few examples of this use case are shown in Figures 11-5 and 11-6.

Figure 11-5. Crop-based art direction example (image taken from Google developers)

Figure 11-6. Art direction that changes the position and size of images between small and large viewports (image is a screenshot from http://kobe-marathon-qbb.com/)

What we see in these examples is that serving the intended images is essential in order for the user to fully understand the page and for the page to be properly laid out. The different proportions of the images mean that if we were to serve images that don’t fit the layout, the layout would break.

In a way, this use case is less about “performance optimization” and more about “content optimization.” The problem here is not so much about the image’s quality as it is about getting the image’s message accross to users, regardless of viewport restrictions. With that said, when large parts of the image are being cropped away when served to mobile devices, that certainly doesn’t help the site’s performance.

Art Direction Versus Resolution Switching

The fixed and variable dimensions use cases are often referred to together as resolution switching. The main difference between resolution switching and art direction is an issue of control. In the resolution switching cases, the issue at hand is quality. We want the user to get the best experience, where experience is a combination of visual quality and speed (and one might claim that the eventual bandwidth costs and battery life are also part of the experience). These aspects of the user’s experience are not something that the web developer knows in advance, and any attempts to predict the user’s “context” are bound to fail.

Therefore, for the resolution switching case, we want to give the browser the final word. Our solutions enable the developer to declare multiple resources and enable the browser to pick the one that best fits the user’s current situation.

On the other hand, for the art direction case, the issue is one of fidelity. If the browser shows a different image than the one the developer intended, users may get a distorted image or a broken layout, hurting their experience as well as their ability to use the site properly. In this case, the browser doesn’t know more about the user’s context than the site’s developer did when creating the site. So, we want the browser to be “bound” to obey the developer’s instructions and download the specific image that the developer had in mind for particular viewport dimensions or other environmental constraints.

We need two distinct mechanisms to handle each one of these cases, one where the control is in the browser’s hands, with the guidance of developers, and another where the control is in the developer’s hands.

Avoiding “Download and Hide”

The “download and hide” scenario often happens when desktop sites are retrofitted to be responsive and some parts of the page are just not needed on mobile.

A common mistake in such a scenario is to hide the unnecessary parts with CSS and hope for the best. The problem with that approach (as you probably guessed from the scenario’s name) is that even if the end user does not see these parts of the page, the resources they require—and content images in particular—are downloaded nonetheless.

You could think of this scenario as a form of art direction, where in some viewport sizes, the required image is a zero-sized one. We’ll later see how to address this use case properly.

Use Cases Are Not Mutually Exclusive

There can be cases where a certain image does not strictly fall into a single use case, but combines a little bit of both art direction and resolution switching, depending on the breakpoint we see it in. For example, consider Figure 11-7.

Figure 11-7. An image that fits into the variable width use case up to a certain breakpoint, then gets cropped

We could also mix the image formats use case here to further optimize the delivery when content negotiation is not an option.

So when we’re looking into making a certain image responsive, the question we should ask ourselves is not “what is the use case this image fits into?” but “what are the use cases?”

srcset x Descriptor

So, you have an image of a cat that you want to display on your site, where said image would have the same dimensions regardless of responsive breakpoints. So, you start out writing something like:

<img src="cat.jpg" alt="A funky cat">

But, when viewing that work from a retina screen, you notice a blur. Each image pixel is displayed over 4 physical pixels, and it just doesn’t look sharp. You want to provide the browser with a 2x image, twice the width and twice the height, which would get rid of the blurriness, but without sending that over to browsers that don’t need it. The following syntax will do just that:

<img src="cat.jpg" srcset="cat_2x.jpg 2x" alt="A funky cat">

That is not extremely different than what we’ve seen earlier. All we added is the srcset attribute, providing the browser an alternative resource to fetch for the same image. That attribute enables the browser to create a list of available resources, so it can pick which one to download and display.

As you probably noticed, that syntax enables us to tackle the fixed-dimensions use case we discussed earlier.

But what if we wanted to provide multiple alternative resources? Well, srcset is actually a comma-delimited list, so you can provide as many resources as you want!

<img src="cat.jpg" srcset="cat_2x.jpg 2x, cat_2.8x.jpg 2.8x" alt="A funky cat">

Simple, right? The value of srcset in the preceding example is a list of the resources for the browser to choose from. Each resource has a descriptor attached to it, which tells the browser something about this resource to make its job of picking the right one easier. In this case, the descriptor in question is the x descriptor, which describes the resource’s density. That gives the browser the knowledge to pick the resource that best fits the user’s screen.

Image density is the ratio between the image pixels that you provide the browser and the area (in CSS pixels) that the image is displayed on. Let’s say you have a 400×400 CSS pixels space in which to fit an image and you provide the browser a 600×600 pixel image to fit that space. That image would be displayed with a density of 1.5, and would look perfectly sharp on screens with up to similar density, but not necessarily on screens with higher density.

Now, different browsers may do different things when picking the best resource, and they are entitled to do that. The specification is purposely vague about the selection process inside srcset, in order to enable browsers to innovate in that space. Therefore, browsers can take not only the screen density into account, but also the user’s cache state, network conditions, user preference, and more.

Already today there are differences between browsers regarding which image they fetch when the screen is not an exact match to any of the resources, or differences when some of the resources in the list are already in the browser’s cache. These differences are likely to increase over time as browsers get smarter about resource downloads, so you should not rely on the browser picking one specific resource over another.

srcset w Descriptor

Now the fixed-dimensions case is great when this is what you need, but in responsive designs the variable-dimensions case is often more common. Your image changes its size as the viewport changes, either due to the fluid layout that contains it or due to a breakpoint change that impacted it.

The syntax to achieve that would be:

<img src="cat.jpg" srcset="cat_200w.jpg 200w, cat_400w.jpg 400w" sizes="100vw"
 alt="A funky cat">

That’s very similar to what we’ve seen before, but with different descriptors. The w descriptor is there to describe the width of the image, in pixels.

And what’s that sizes attribute that I sneaked in there, you ask? Nice catch—I hoped I could get that by you. It’s there to tell the browser what the image’s display width would be. The browser needs that in order to figure out which resource it should download, and it really doesn’t know that at the time it starts downloading images.

In order to know the dimensions in which images will be displayed, the browser needs to perform layout, and in order to do that it needs to download all the CSS in your page, process it, and calculate which rules apply. Only then can it calculate the layout of all the elements in the page, and it’s pretty late in the game. To make things worse, the downloaded image resources can also impact the layout, as the image’s intrinsic dimensions are used to lay it out in case neither HTML nor CSS knows better. So, waiting for layout information to decide which image to download is just not an option.

This is the reason we need the sizes attribute to tell the browser what the image dimensions would be. In the preceding case we told the browser that the images will be displayed at 100vw or 100% of the viewport width (see Figure 11-8).

Figure 11-8. An image that takes the full width of the viewport

100vw is also the image width that the browser would assume when calculating density if we didn’t include a sizes attribute at all (even though we probably should include it, as our markup would be invalid if we didn’t).

But often in responsive layouts, images take a smaller chunk of the viewport, and assuming they are 100% of the viewport width would mean downloading images that are just too large. For these cases, we can define a different CSS length as the value of sizes—for example, 70vw (see Figure 11-9).

Figure 11-9. An image that takes only part of the viewport

See? That wasn’t so hard, was it?

Variable width images often require viewport-dependent CSS lengths (e.g., the vw units), but if we want to, we could satisfy the fixed-dimensions case by using w descriptors and setting sizes to a non-viewport-dependent length—500px.

In other types of layout it can get more complicated than that. How can we tackle images that “shrink” inside the page’s responsive breakpoints, but change dimensions entirely between breakpoints, as in Figure 11-10?

Figure 11-10. A two-column layout that turns into a single-column layout; such cases result in a nonlinear relationship between viewport dimensions and image dimensions

We can handle that case by extending the sizes value beyond the simple CSS length, including the various breakpoints. The syntax to implement that would be:

<img src="cat.jpg" srcset="cat_200w.jpg 200w, cat_400w.jpg 400w"
sizes="(max-width: 460px) 50vw, (max-width: 1200px) 25vw, 300px"
alt="A funky cat">

How does that work? The browser takes the entire value of the sizes attribute, and breaks it up into pairs of a media condition and a CSS length. A media condition is very similar to a media query, only it doesn’t specify a media type and is optional. The browser goes over the media condition and length pairs, and checks if the media condition matches or is missing. If so, the CSS length is picked to be the element’s source size. The browser uses the source size in order to translate all the w descriptors into internal density values, and then applies the same algorithm that it applies on x descriptors, so again, it takes into account the screen density and other factors when picking the appropriate resources from srcset.

It’s important to emphasize that sizes is an optimization and even if you define a rough sizes value (e.g., 100vw), it is still in many cases better than simply sending the same image to tiny devices and retina 28-inch displays. But sizes enables you to get as close as you’d like to the actual display dimensions, and enables the browser to pick the right image resource accordingly.

<picture>

The previous srcset descriptors all assume that all the image resources represent the “same” image and are interchangeable, only in differing qualities and dimensions. While that’s enough to cover the fixed-dimensions and variable-dimensions use cases, when it comes to art direction, we need more control than that. We need to be able to tell the browser “download this image resource in this particular breakpoint” and be fairly confident that it will. Otherwise, layout may break and our site may become unusable.

So, how do we define the image resources for Figure 11-11?

Figure 11-11. An art direction example

How does that work? The important bit to understand is that even though <picture> gets a lot of attention, <img> is still the element that drives image loading and display. Among other things, this means that if <img> were missing from the preceding example, nothing would be displayed. So, <img> gets created by the parser, and then the element checks to see if it has a <picture> parent before it starts loading an image. If a <picture> parent is present, the <img> element walks that parent’s <source> children until it reaches itself, and picks the first one that matches as the source of image resources. In our case, “matches” means that the media query inside the media attribute matches. If no <source> element matches, <img> will be its own resource source.

Once a source is picked, its srcset attribute will be used to pick the right resource, in a similar process to what we’ve seen earlier.

And while we need strict control in order to get art direction right between our responsive breakpoints, we may also need to be able to mix that with the other use cases inside the breakpoints.

That can be achieved with syntax like the following (see Figure 11-12):

<picture>
    <source media="(max-width: 600px)"
    srcset="narrow_viewport_200.jpg 200w,
    narrow_viewport_400.jpg 400w">
    <img srcset="wide_viewport_200.jpg 200w,
    wide_viewport_400.jpg 400w" alt="probably a cat">
</picture>

Figure 11-12. An image that fits into the variable width use case up to a certain breakpoint, then gets cropped

Another use case we talked about earlier is the “download and hide” case. We can resolve that using <picture> by adding a “spacer GIF” data URI as the source for the images that are not supposed to be there. The reason this is neccessary is that the selection algorithm skips a <source> without any srcset attribute. So we need our <source> to have a valid srcset, only with a meaningless small image.

So, if we want our image to “disappear” at viewports smaller than 600 pixels, we could do:

<picture>
    <source media="(max-width: 600px)"
    srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAA
    AALAAAAAABAAEAAAIBRAA7">
    <img srcset="image_only_needed_for_wide_viewports.jpg">
</picture>

Serving Different Image Formats

The main reason the RICG came up with <picture> was to tackle the art direction use case, but that’s not the only thing it is good for. Serving different image formats to browsers that support them while having a common-ground image format as a fallback is another use case it tackles.

If we look at font, video, or audio files, the web platform supports many different formats and enables client-side fallback for them right from the moment that these media types are added to the Web. Contrary to that, images always lacked such a client-side fallback mechanism. As <img> stabilized support on the three major file formats before the first browser wars were over, there were no compatibility issues related to image format support, so no one worked on a mechanism to enable them. When new file formats such as WebP or JPEG XR were introduced, the answer to people trying to use them was content negotiation, and modifying the server’s logic so that it would return the newly supported types only to browsers that supported them.

That is, until responsive images became a thing. While the RICG was working on new markup solutions to load responsive images, it became clear that the same constructs (with slight additions) could be used to create a client-side fallback, and help introduce new image formats to browsers that supported them, even if you had no control over the server’s logic.

How can we do that, you ask? By using the type attribute!

<picture>
    <source type="image/webp" srcset="rad_wolf.webp">
    <source type="image/jxr" srcset="rad_wolf.jxr">
    <img src="rad_wolf.jpg" alt="A rad wolf">
</picture>

Here again, the browser will go over the list of <source> elements and pick the first that matches, only this time, “matches” means that the type attribute contains a MIME type the browser supports.

To Picturefill or Not to Picturefill, That Is the Question

The first question that often comes up when discussing these features is: “do we need to polyfill it for legacy browsers?” The answer, like many things in computer science, is: “it depends.”

The features were built with an inherent fallback in mind. As <img> must always be present in the markup, it is sufficient to add a src attribute to it with a fallback image, and nonsupporting browsers will have something to display. In many cases, that’s good enough.

Until recently browser support for the entire set of responsive images features was not complete, and therefore if you needed art direction, for example, you had to use a polyfill, whereas if you needed fixed-width resources, you could have gone with a reasonable fallback instead.

But since then, support was added to all major browsers, and nowadays the only reason to use a polyfill is if you need to support older browsers, such as IE, and such support cannot be accomplished with a simple fallback image, since the image is art-directed.

If you do need to use a polyfill, the official and standard-compliant polyfill would be Picturefill.

Intrinsic Dimensions

Every image has intrinsic dimensions that are defined by the image file itself and composed of the image’s width and height in “physical” pixels. The browser then takes these intrinsic dimensions into account when it decides how to lay out the image. If there are no HTML- or CSS-based instructions that tell the browser what the image’s display dimensions should be, it is the intrinsic dimensions that dictate that.

However, when we’re talking about responsive images, we don’t want them to be displayed according to their “physical” intrinsic dimensions, but according to dimensions adjusted to the image’s density. That helps us get properly dimensioned and sharp images, rather than oversized blurry ones—for example, when sending 2x images to the browser.

How does the browser accomplish that? When picking image resources as part of srcset’s selection algorithm, the browser calculates the image density. If the image has an x descriptor, that’s easy. The descriptor’s value is the value of the image’s density. If the image has a w descriptor, we also need to know which dimensions the image will be displayed in. As we’ve seen earlier, that’s where the sizes attribute comes into play. The browser takes the sizes attribute, figures out the CSS length that applies to the current breakpoint, and uses that and the resource’s w descriptor in order to conclude the image’s density. The image density is then used by the browser to compensate and correct the “physical” dimensions when it calculates intrinsic dimensions.

OK, but why do I think that this long and complicated story is of interest here? Isn’t that the problem of the people working on the browser implementation?

Well, it has real-life implications because if you give the browser the wrong details, the outcome may surprise you. I’ve seen many examples where people put in approximate w descriptor values or incorrect sizes attribute values while relying on the image’s intrinsic dimensions, and end up surprised that the image is displayed in the wrong dimensions.

The point is, if you feed the browser the wrong data about the images you give it, you’re likely to have a bad time debugging why your images are too big or too small. So don’t.

Selection Algorithms

We already talked about the ways that the browser selects the right <source>, the right sizes length, and the image resource in srcset, but it’s worth repeating in order to avoid confusion.

<source> elements are picked using a “first-match” algorithm. The first <source> element that matches both media and type (where a missing attribute is considered as matched) is the one that gets picked.

For sizes, it is very similar. The browser goes over the list of media conditions and CSS length pairs, and picks the first length with a matching media condition, or one with no media condition at all. This is why we often leave the last length value in sizes as a standalone value, to be used as a fallback.

srcset, on the other hand, doesn’t use a “first-match” algorithm, so the order doesn’t matter unless you have resources with the same density. That means that as long as your descriptors are correct, you don’t need to worry about the order of the resources inside srcset.

srcset Resource Selection May Change

srcset was designed so that its selection algorithm can evolve over time to make smarter decisions about the tradeoff between image quality and download speeds. We want srcset to be able to respond to many things beyond simple screen density: browser cache, bandwidth conditions, user preferences, network costs, and other user conditions. Therefore, and since srcset only covers cases that are not related to art direction, the browser has a great deal of liberty when it comes to picking resources.

That’s the reason you shouldn’t rely on what you think the browser should load in different conditions when using srcset. That can and will differ between browsers and between browser versions. As browsers improve and get smarter, your assumptions regarding their behavior will not remain accurate for long. So make no assumptions and let the browser do its thing.

Feature Detection

Since the responsive images features were recently introduced to the web platform, there is a chance that you’ll need to be able to tell if your user’s current browser supports these features or not. The classic use case for this is when creating a polyfill (which you don’t really need to do, as Picturefill is well maintained and fully supports the syntax), but there could be other occasions where you find yourself trying to figure out browser support for the responsive images features.

For these cases, you can use the presence of HTMLPictureElement in window to detect support for <picture> and use the presence of sizes and srcset in an HTMLImageElement node to figure out if they are supported.

More concretely, here’s how Picturefill detects that support:

var image = document.createElement( "img" );
var srcsetSupport = "srcset" in image;
var sizes support = "sizes" in image;
var pictureSupport = !!window.HTMLPictureElement;

currentSrc

One more tool you can use when working with responsive images is the currentSrc property on HTMLImageElement, which enables you to see which resource is currently loaded and displayed on a specific <img> element.

You can use this if, say, your JS interaction with an art-directed image should differ in case a different resource is picked.

Background Images

In the early days of responsive images, the subject of background images wasn’t considered important, and the focus was on content images. After all, background images could be controlled through media queries, and on top of that, WebKit-based browsers (so, Safari and Chrome) supported the -webkit-image-set CSS property, which enables the browser to load based on device pixel ratio (DPR). That was considered a handy shortcut to spelling out separate rules based on the resolution or -webkit-device-pixel-ratio media queries.

So we continued to resolve content images, leaving background images as they were. Only now, with the hindsight of the solutions for content images, do we see the deficiencies that still need to be resolved in background images.

So, let’s take a look at how each one of the use cases can be resolved for background images.

The fixed-width images can be resolved with a fairly simple markup (which resembles, and in fact inspired, the markup for srcset’s x descriptor):

.fixed-dimensions-image {
    background-image: -webkit-image-set(url(1x.jpg) 1x, url(2x.jpg) 2x);
}

The standard form of that is the unprefixed image-set. Unfortunately, that is not implemented anywhere at the time of this writing.

The art direction use case is easy to solve with media queries:

.art-directed-image {
    background-image: url(narrow.jpg);
}
@media screen and (min-width: 800px) {
    .art-directed-image {
        background-image: url(wide.jpg);
    }
}

But there’s no way today to define a background image that loads an efficient, variable-width image. One could imagine an extension to image-set that includes something like the w descriptor srcset has, but that’s not yet specified or implemented.

Height Descriptors

You may have noticed that when discussing the use cases, we’ve used the term variable dimensions, yet the only resource descriptor we have for this case is the w descriptor, describing the resource’s width. Images are two-dimensional! That’s not fair!

While we were working on the responsive images solutions, we noticed the same injustice, yet the major use case to tackle was solving width-constrained layouts. We had significantly fewer examples for height-constrained layouts, so we preferred to wait with that use case until there’s more experience with it “in the wild.”

Nevertheless, the processing algorithms take the future existance of an h descriptor into account, and ensure that the introduction of such a descriptor will go over smoothly.

After having the basic set of features out there, with developers using them in production, we now see some demand for height-constrained layouts, mainly for image galleries. So, hopefully work on that front can continue and h descriptors will eventually be part of the srcset.

Progressive JPEG

As we’ve seen earlier in the book, progressive JPEG is, well, progressive. The browser can decode it as it is being downloaded, and the result is a full image, with its details filling in as more chunks of image data are downloaded.

Therefore, we could emulate a lower-resolution image (or a smaller image) by truncating a high-resolution, large image and scaling it appropriately. Assuming we have multiple JPEG scans, we could use something like SSIM to determine the appropriate size and resolution for each one of the scans, and then communicate that information to the browser (e.g., using special JPEG markers at the start of the file), and have it download only the scans that it needs.

Such a concept has been experimented with in the past. It seems like something that might work for the fixed- and variable-dimensions cases, but not for the art direction use case.

Additionally, from experimentation, if the images that you’re trying to serve need to fit both a very small space (e.g., on low-end devices) and a very large one (e.g., retina 28-inch screens), quality will suffer or bytes will be wasted. There’s a limit to the range of quality that can be communicated using progressive JPEG scans.

JPEG 2000

JPEG 2000 (which we discussed in Chapter 5) is a progressive format by nature, and therefore, at least in theory, could be an ideal candidate for progressive loading.

Unfortunately, previous experiments that were conducted on that front proved it to be less promising in practice.

Responsive Image Container

There have also been attempts (by yours truly) to create a responsive image container—an image-format-agnostic container that would encode the image using different layers, where the first layer is a thumbnail of the image, and each consecutive layer adds more information to the image, enabling it to target higher-resolution screens, larger display dimensions, and even crop-based art direction. The intent behind creating a container rather than a full-fledged new format was to avoid patent and political issues often surrounding file formats, likely increasing its chances for adoption.

FLIF

More recently, a lossless progressive file format named FLIF was introduced. The name stands for Free Lossless Image Format, and it shows very good results when compared to other lossless formats.

One of its most touted advantages is its progressive nature, which could make it a candidate for a responsive format. However, it’s still very early days for the format, so it’s hard to be certain of the direction in which it will evolve. Its lack of a true lossy mode makes it less applicable to real-life imagery than other formats.