Tuesday, November 15, 2016

Playing With The Lytro Illum 3: Processing Software and Stills

I found the Lytro Desktop software difficult to deal with, especially compared to established products like Lightroom or even Photoshop. And a lot of my critiques really come down to "Desktop isn't Lightroom" so decide for yourself if you think that's fair.

First off, "Desktop" is just the name of Lytro's software.

System Requirements

Start with the system requirements. The minimum requirements for Desktop 4 on a Mac are Yosemite(10.10, 2014) with 4GB RAM. Since they have an unusual proprietary format for their images, I have no choice(really) but to use their Desktop software.

I have a very old MacBook Pro running Snow Leopard(10.6, 2009) (and I can still run Lightroom 4 on it). This was almost a non-starter. Fortunately, a friend let me use her MacBook Air (Yosemite, 2GB RAM). The memory is a little under the requirements, but it seemed to mostly work with some coaxing. Sometimes it would just be really slow.

Lytro also has Desktop 5, but it is only available on PCs.

To me, it looks like Lytro's focus is in exploring new techniques to optimize processing the images(things that would be applicable to VR and cinema applications), and supporting older operating systems limits that - . It makes sense that they would research hardware solutions, and a Mac may limit the ability to swap graphics cards.
I think I understand it, but it doesn't mean I have to like it.

Disk space is another issue. The images are about 50MB each for a 4 Megapixel/40 Megaray image. By comparison the Canon 5Dmk2 images are about 12MB each for a 21 Megapixel image.

Importing into Desktop

The first thing I was frustrated with was the catalog management in Desktop:
  • The load times were pretty slow - approx. 4 min 50 sec to copy and process the image. I regularly shoot 100-200 images in one batch so that would be 8 hours/overnight. These are large files, so it's somewhat understandable, but still something I have to factor in evaluating the workflow.
  • If I load more than 70-100 images at a time, the software hangs. I think it ran out of memory so I need to break up the set into smaller groups of 70-100.
  • If Desktop freezes and I quit and start again, I cannot pick up from where I left off. I have to start over. In Lightroom, I can choose which individual files to load. In Lytro Desktop, I can only choose entire directories.
  • I would frequently mix up the commands to create a new library/catalog vs. import a new directory into the current library.
Directory listing for one image (raw is the main image)
On the memory card, the raw images have relatively readable names like IMG_1049.LFR. When Desktop imports an image into the catalog, creates a directory with an incomprehensible hash like fa461a11-b6fe-4924-87fc-c3764cac02b2 for a name, copies the raw image file (raw.lfp - 50Mb), and creates a few sidecar files.

Between the directory name and the file names, there is no indication which image this came from. If the import was interrupted, there is no easy way to tell which images have successfully imported, and which still need to be added. Also, if I delete images while working inside Desktop, I have to write down the image numbers if I want to delete the originals too. I can't just look on the file system to sync them up.

Desktop keeps a database for the images (similar to a Lightroom catalog). I'm not sure what it keeps track of(probably the adjustments). As an example, the database is 60.4k bytes for 44 images, which isn't too big. (The Lightroom database/catalog can get very large, especially if you use brush adjustments.)

Lightroom gives me the choice of copying the file(good if I'm importing from a memory card) or leaving it in place(good if I've already copied it on disk). Desktop doesn't offer this choice and always makes a copy, but changes the name to something incomprehensible.

I want my image files to have the original numbers so I first copy the images from the memory card onto disk. Copying files from memory card to hard disk isn't too bad - approx 3.2 sec/image - but it does mean that I'm keeping 2 copies of the image. 

With a Canon 5Dmk2, I could quickly scroll through a lot of photos on the camera back. Advancing to the next photo is instant. With the Lytro Illum it takes a second or so to advance, which makes it difficult, if not impossible to review the whole batch on-camera. Also, I think that previewing the image on-camera saves the preview on the card, taking up valuable space.

In Desktop, I kept each small group as a separate library/catalog, which was a little cumbersome. But I just didn't trust Desktop to manage more than 100 images. It's possible that one catalog could have handled more that one directory, but I was running out of patience and didn't try it. It was unclear to me if Desktop tried to keep a cache of all the images in a catalog.

UI Layouts and Usability

Modes of Desktop (currently showing Library mode w/Thumbnails)
The layout is controlled in two parts.
In the upper right, there are three modes:
  1. Thumbnail
  2. Full screen
  3. Full screen w/Filmstrip
These are visible for all of the main modes (on the far left column), so they appear to be independent.
But really, not every combination was valid, so it really should have been been a sub-menu (like it is in Lightroom)

In Library mode, you can use any view (Thumbnail, Full screen, Filmstrip). The Library management felt clunky.
  • For one thing, I cannot filter based on the stars(like "2 or higher").
  • I'm used to a 5 star scale (like Lightroom and iTunes). Desktop uses a 3 star scale.
  • When I import from a directory (the only choice), it creates an album. I can also create arbitrary albums, and a given photo may be a member of multiple albums (similar to being able to be in multiple collections in Lightroom).
  • In thumbnail mode, I cannot see the file name.

Desktop in Info mode w/Thumbnails
In Info mode, I can also use any view (Thumbnail, Full screen, or Filmstrip).

In Lightroom, most of this information is available as an overlay or as a part of Library. In Desktop, it is a separate main mode.
Desktop in Animate mode
In Animation mode, it switches the main view to a special animation layout with a time line.

(more on this in an upcoming blog post)
Desktop in Adjust mode w/Filmstrip
Adjust mode is where I spend most of my time in Desktop.
Desktop in Adjust mode w/Thumbnails - I don't think this should be possible!
It is possible to set Adjust to the Thumbnail view mode, but I don't think it should be - I cannot make any adjustments.  (In Lightroom, I actually can edit a batch of photos in the thumbnail layout)

Desktop doesn't do anything to prevent this combination.

When I'm reviewing my photos in Lightroom, I can advance to the next photo nearly instantaneously. Desktop takes about 6 seconds to step to the next/previous photo (in both Library and Adjust mode) so I cannot use the workflow that I'm used to and it is difficult to flip back and forth to compare similar shots.

Desktop did seem to give OSX some preview capability, so I would try to make my quick reject/delete decisions in OSX Finder in Coverflow mode before creating a Desktop library. Once I load the photos into Desktop, I'll  look over them in thumbnail mode to try to reject some of them.

"Undo" functionality is better in Lightroom. It is more reliable and shows me the whole undo history. Before I undo, I know what operation I will undo.

In Desktop, sometimes if I moved a slider, I could not undo it. But if I changed the text box next to it, it was undo-able. Later, I was able to undo from either slider or textbox. Later, I was unable to undo from either slider or textbox. Desktop does not show the undo stack and does not tell me what the last operation was.

I cannot type values in for the refocusing. I can move sliders and I can click sample points in the image. But I cannot get the precision of just telling Desktop how far out I want it to focus.

A regular camera has one focal plane. Desktop lets you define a wider range for being in focus. Though I personally don't like using this feature - the gimmick with the Lytro images is to see things shift in and out of focus and I don't want to expand this range.

Lytro focused its efforts on making the refocusing fast, which is good. But it was at the expense of all other operations (color temperature, exposure, cropping, ...). In Lightroom, these are handled as metadata and cached in a preview image. In Desktop, making any color, exposure, or cropping adjustment would trigger reprocessing the image. I made a few in a relatively new session and they took around 42-45 sec each to process.

Exporting Still Images

The Desktop export options are limited when I want to export a batch of photos:
  • Original name with original number
  • New name with new number (sequential starting from 1)
In Lightroom, I often export with a new descriptive basename (something other than IMG), but keep the original number so I can more easily find the image for follow-up edits.

Also, Desktop only lets me export at full size. Lightroom lets me resize to any dimension. Though in Desktop's case, the image is already small (4 Megapixels), so I might not want to shrink it much anyway. Also, there are no options to add watermarks.

Using Depth Maps

In Part 1, I described what I think were the basics of refocusing and my point there was that I thought the light field was sufficient to reconstruct the refocused image with a change in aperture and even add special effects like tilt shift or camera offset. I don't think I need to know anything about what is in front of the camera(i.e. the depth map) to do the ray trace. But I understand that could be slow to compute.

(My speculation) It looks like Lytro/Desktop reorganizes the data to try to recreate the scene in front of the camera:
  1. Use the raw light field to compute the scene with maximum sharpness and a depth map. This is a one-time happens when the raw file is imported and processed.
  2. When viewing/editing the file, use the depth to drive the depth of field defocus.
With an accurate depth map, there is potential to derive more information about the scene (such as normal maps). In the last entry, I described a basic algorithm to find depth maps, and some of the vulnerabilities.

Something else that I find confusing in the UI is that the focus point is some arbitrary value from -6 to +6 (I think - maybe -4 to +4). But there's no explanation anywhere what these units mean in relation to a distance from the camera (like inches, feet, centimeters...). It feels like it is just trying to separate planes and apply some arbitrary blur. It is unclear how the amount of blur relates to the focal length and actual distance from camera. I'm uncertain if they are even defocusing. (They might be gaussian blurring - the final look appears more smeary to me.)

Editing the Depth Map

The depth maps I've gotten are really inaccurate. I keep harping on this, but I really shouldn't have to rotoscope and repaint the depth map myself. I think that's basically what the technology of the camera was promising to do. It feels like they are just throwing away light field data and asking me to replace it with my aribitrary painting. This doesn't scale for large events and I don't have the time for this.  Desktop does provide ways to edit the depth:
  • They can be exported to an external editor of my choice (like Photoshop). 
  • Desktop provides an internal depth map editor.
The internal depth map editor is difficult to use.

First, I had an intermittent problem that the "Done" button was greyed out, so Desktop wouldn't let me save my changes. But luckily, that wasn't all the time. I couldn't reliably reproduce the bug though.

For example, here is one of the photos from Inspiration Weekend and the depth map.

There are a lot of problems with the depth map:
  • Hyunjung in the foreground is the closest part of the scene, so she should be the darkest part of the depth map. Though the grey is the darkest part of the depth map, it's not really that dark(probably more than 50%). There is a very compressed dynamic range, which makes it harder to emphasize the separation.
  • Hyungjung's shirt and feet are (approximately) the same distance away from the camera as her pants. But in the depth map, most of her pants were calculated to be nearly white - the farthest distance away.
  • The floor next to Hyunjung's feet is bright white (far away). Close to her feet, the floor should be as dark as her shoe, and it should have a smooth gradation going back. The stage shows up as darker (closer) than the wall in front of it.
  • In the background, Soochan shows up as brighter (farther) than the wall behind him.

If I wanted to push the separation, I could try an operation like "Histogram Equalization" which tries to use the full dynamic range.

But this feels very artificial to me, and just emphasizes the idea that the -6 to +6 numbers in the depth map seem to have no physical meaning or relationship to a real world depth measurement.

Also, the histogram equalization seemed like it was undo-able so just to take a peek at it, you probably should make a copy of the photo.

The internal editor had inaccurate color sampling.
For example, I want to repair the depth of the pants. So I want to get the paint value/depth from the value/depth of the boots with the eyedropper tool and it gets the value 196.
But I start painting and it looks like my paint stroke is a little darker than the boot. When I sample this with the eyedropper tool, I get the value 183.

It's like there is a problem with the lookup table(LUT). It's unclear whether the values are stored as linear or LUTted. It's unclear whether the eyedropper tool is reading the LUTted or non-LUTted value. It's unclear in which color space the paint is being applied.

But the end result is that I can't accurately sample a value, which does pretty much render the depth paint tool useless.

It can still be repaired in Photoshop (or any external editor), though I haven't actually verified that. But the internal tool is broken.

Resulting workflow

Overall, I was very frustrated trying to edit photos in Lytro's Desktop software. Color, cropping, and exposure adjustments took forever. Though the refocusing generally didn't take too long.

I'm used to editing photos from my Canon 5Dmk2 in Lightroom. It's a clean way to organize my photos and is great at color/exposure adjustments, cropping, and other ways to artistically shape the photo. But Lightroom doesn't have access to the refocusing and light field information.

First, when evaluating photos, I found that OSX's Finder could actually give me a large preview in coverflow mode, so I could have some idea whether the pose was any good, and decide whether to delete the photo or import it into Desktop. It is expensive(time, disk space) to import into Desktop, so if I can filter out some images ahead of time, it's worth it.
By contrast, Lightroom moves fast enough that I tend to do fewer deletes on-camera and just import everything into Lightroom and evaluate it there.

So overall, I try to do as little as I can in Desktop.
  • I may do some batch edits for exposure and rough color temperature to put it in the neighborhood. Once I export the photo, I won't have access to the raw data anymore, so I will only get limited color temperature adjustments afterward.
  • I do have to choose the f/stop and the focus point(or focus range). Generally, this goes pretty quickly. Usually I have something in mind for the f/stop of the whole set, and there are a couple outliers that I need to have a deeper focus range. There are realistically only a few points I can focus on in the image.
Once I had the basics set in Desktop, I exported all of the images to finish off in Lightroom.

Then in Lightroom, I would make the more refined decisions about exposure, boosting the shadows, muting highlights, fine adjustment of color temperature and tint, cropping, setting the saturation, adding gradients if necessary, ... basically any of the artistic and composition decisions.

No comments:

Post a Comment