Playing with Jupyter and Darktable db… which focal lens do I like more?

I am trying to embed a full Jupyter notebook here… it should be more or less auto-explicative. Please comment if you need help!

Ok, so now: what new nice prime lens should I buy?

 

explore_data

Exploring your DT database

Copy (for safety) your db:

cp $HOME/.config/darktable/library.db dtlibrary.db

These are the available tables:

    sqlite> .tables
    color_labels     history          mask             tagged_images  
    db_info          images           meta_data        used_tags      
    film_rolls       legacy_presets   selected_images

First of all, you load the database and save the tables you need in csv format. See for example http://www.sqlitetutorial.net/sqlite-tutorial/sqlite-export-csv/

    sqlite3 dtlibrary.db
    SQLite version 3.22.0 2018-01-22 18:45:57
    Enter ".help" for usage hints.
    sqlite> .headers on
    sqlite> .mode csv
    sqlite> .output images.csv
    sqlite> select * from images;
    sqlite> .quit
In [1]:
# sources
# https://songhuiming.github.io/pages/2017/04/02/jupyter-and-pandas-display/
#
import sys
import os
# BEWARE only for command line interactive usage
from math import *
import scipy as sp
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
pd.set_option('display.height', 10)
pd.set_option('display.max_rows', 8)
pd.set_option('display.max_columns', 10)
pd.set_option('display.width', 80)
In [2]:
allphotos = pd.read_csv("images.csv")
In [3]:
allphotos
Out[3]:
id group_id film_id width height write_timestamp history_end altitude position aspect_ratio
0 8244 8244 295 5344 3516 1440610685 13 NaN 35407710388224 0.0
1 8245 8244 295 5344 3516 1440610554 15 NaN 35412005355520 0.0
2 8246 8244 295 5344 3516 1440610590 15 NaN 35416300322816 0.0
3 8513 8513 304 4928 3280 1451331582 10 NaN 36563056590848 0.0
5653 17653 17654 564 5184 3888 1565097337 0 NaN 75797582839808 NaN
5654 17654 17654 564 5240 3912 1565097337 0 NaN 75801877807104 NaN
5655 17655 17656 564 5184 3888 1565097337 0 NaN 75806172774400 NaN
5656 17656 17656 564 5240 3912 1565097337 0 NaN 75810467741696 NaN

5657 rows × 42 columns

In [4]:
allphotos.columns
Out[4]:
Index(['id', 'group_id', 'film_id', 'width', 'height', 'filename', 'maker',
       'model', 'lens', 'exposure', 'aperture', 'iso', 'focal_length',
       'focus_distance', 'datetime_taken', 'flags', 'output_width',
       'output_height', 'crop', 'raw_parameters', 'raw_denoise_threshold',
       'raw_auto_bright_threshold', 'raw_black', 'raw_maximum', 'caption',
       'description', 'license', 'sha1sum', 'orientation', 'histogram',
       'lightmap', 'longitude', 'latitude', 'color_matrix', 'colorspace',
       'version', 'max_version', 'write_timestamp', 'history_end', 'altitude',
       'position', 'aspect_ratio'],
      dtype='object')

Fix the database

A lot of entries for OLYMPUS are bad. I fix this here, but you can fix the database also with

check: select * from images where maker like "%OLYMPUS%" and crop=0;
change: update images set crop=2 where maker like "%OLYMPUS%" and crop=0;
In [5]:
allphotos.loc[(allphotos.maker=="OLYMPUS CORPORATION")&(allphotos.crop==0), 'crop']=2
In [6]:
allphotos['FLeq35'] = allphotos.crop * allphotos.focal_length
In [7]:
FLseries = allphotos[allphotos.FLeq35>0].FLeq35
In [8]:
FLs=np.array(FLseries)
In [9]:
FLs
Out[9]:
array([  33.        ,   33.        ,   31.99999899, ...,  250.        ,
        140.        ,  140.        ])
In [10]:
plt.hist(FLs, bins =[0,20,40,60,80,120,200,500])
Out[10]:
(array([   21.,  3007.,   971.,   504.,   603.,   223.,   316.]),
 array([  0,  20,  40,  60,  80, 120, 200, 500]),
 <a list of 7 Patch objects>)

Let’s try to separate them based on camera

In [11]:
cam_and_fl = allphotos[allphotos.FLeq35>0].loc[:, ['model', 'FLeq35']]
In [12]:
cam_and_fl
Out[12]:
model FLeq35
6 SLT-A55V 33.000000
7 SLT-A55V 33.000000
9 DMC-LX5 31.999999
10 SLT-A55V 45.000000
5653 E-M1MarkII 250.000000
5654 E-M1MarkII 250.000000
5655 E-M1MarkII 140.000000
5656 E-M1MarkII 140.000000

5645 rows × 2 columns

In [13]:
models = cam_and_fl.model.unique(); models
Out[13]:
array(['SLT-A55V', 'DMC-LX5', 'DMC-TZ100', 'TG-870', 'E-M5MarkII',
       'E-M1MarkII', 'E-M10 Mark III'], dtype=object)
In [14]:
from matplotlib.ticker import ScalarFormatter
for i, mymod in enumerate(['SLT-A55V', 'DMC-LX5', 'DMC-TZ100', 
       'E-M1MarkII']):
    fig, axs = plt.subplots(nrows=1, figsize=(5,3.5))
    thisfls=cam_and_fl[cam_and_fl.model==mymod].FLeq35
    L=np.array(thisfls)
    axs.set_xscale("log")
    axs.xaxis.set_major_formatter(ScalarFormatter())
    axs.set_title(mymod)
    axs.set_xticks([15, 24, 35, 50, 70, 100, 140, 200, 300])
    axs.hist(L, bins =np.logspace(np.log10(10), np.log10(400), 20 ))

Notes on upgrade from Ubuntu 14.04 to 16.04

I know that the standard policy in Ubuntu is to offer the upgrade from an old LTS to a new one when the new LTS reaches .1, i.e., 16.04.1, but well…

I upgraded my 14.04 in several machines, and I can signal these almost show-stopper problems (workaround exists for all of them, though):

on […]

xkcd color list for LaTeX users…

After reading the (very) fun post about colors name at xkcd, I decided to create a small package to add the color list to my LaTeX documents…

After circumventing a small bug in xcolor, and ten lines of python, I have managed to create a small package that I am sharing here.

To use […]

A new nice trick for the Linux keyboard: greek letters

New layout for my Spanish keyboard

I do not know you, but me (as an engineer) use Greek letters like α, β, γ and so on quite often. I did a small change to my linux keyboard layout: the new part in the keyboard symbols files now is

partial alphanumeric_keys xkb_symbols “standard_tlde” { include “es(basic)” key <TLDE> { [ grave, asciitilde, […]

Eclipse

hearth-shaped eclipse

I suppose that everyone and their cat has eclipse shots this days… but I think this is a bit different:

I was shooting the March 20, 2015 eclipse (in Madrid) through a tree, and as a test I tried to focus on the tree silhouette instead — and that was the result. The normal, […]

Managing PPAs in Ubuntu: a couple of useful scripts.

PPAs are really a great thing in Ubuntu: they allow you to have quite recent programs without upgrading all your system; most Ubuntu users use them extensively.

They have just one problematic side — the havoc that can happen on an upgrade. You see, when you upgrade your system, PPAs are disabled, but the program […]

Quick fix xfig bug

Ok, so I am a computer dinosaur. I still use xfig for much of my drawings, although I recon it is a program from the old century…

Unfortunately, after the recent upgrade to Ubuntu 14.04, a nasty bug surfaced in xfig: a crash whenever you try to draw point-dashed lines.

Fortunately, the launchpad user madHamish […]

The perfect lens kit…

…does not exist, of course. But I think I have found mine; at least for my balance (as an amateur) among portability, flexibility and quality.

I am an amateur photographer; I do not sell photos (apart from some wet dream), and my needs are to produce photos for my family in the form of a […]

How to modify a keyboard layout in Linux

New layout for my Spanish keyboard

Or: how to define a programmer-friendl(er) Spanish keyboard. …and you can add Greek, too!

A bit of history

There was a time when the computers all had the same keyboard — basically, the US one. You really do not need any more; the set of characters that could be represented where, after all, just the […]

Shepherding at (very) small scale

Ants shepherding their "cows"

I recently bought a (wireless) flashgun, a Nissin Di622 mark II for my Sony. And tried the Vivitar 100F3.5 Macro with a method I read from Robin Wong: I used the flashgun hand-held and the camera on the other hand. Firstly. Then I put the camera on a tripod, but with the flash still hand-held, […]