dave 6391ffb860
All checks were successful
Gitea/photolib/pipeline/head This commit looks good
fix thumb service url
2025-02-07 14:45:27 -08:00
2024-12-25 19:31:31 -05:00
2025-02-07 14:45:27 -08:00
2025-02-02 20:16:57 -08:00
2025-02-02 20:16:57 -08:00
2023-01-18 20:05:08 -08:00
2024-11-10 09:37:53 -05:00
2018-09-09 16:45:51 -07:00
2023-01-20 16:04:27 -08:00
2021-08-15 19:04:55 -07:00
2021-08-15 19:04:55 -07:00
2025-02-04 15:09:21 -08:00
2025-01-29 11:39:39 -08:00
2024-12-25 19:31:31 -05:00

Photolib

Cloud-native photo and related media library software, with a focus for photography hobbyists.

Photolib provides a simple, rich, and modern UI to manage your photo and video library. It can scale to host hundreds of thousands of photos but is also light and simple enough to host just the couple photos you shoot with your smartphone.

Photolib stores images as a logical "photoset". This allows identical images and metadata to be grouped into one item. Many photographers shoot in multiple formats - that is, to produce multiple files per click of the shutter. A JPEG and a RAW of some sort (Like Canon's CR2) is common. The primary idea behind Photolib's organization is that one click of a camera shutter equals one photoset.

When photos are imported - assuming they are done in the same batch - files are grouped based on their name and extension, using rules that apply to almost every camera format. For example, a camera shooting under the "multiple formats" mode described above would create, from one shot, IMG_1234.JPG and IMG_1234.CR2. Photolib would detect that since these two images have the same file name (the IMG_1234 part), they are the same photo and should be grouped into a photoset.

In addition to controls in the web interface, Photolib provides both a REST API and command-line tools to manage your photo library.

Installation

Pre-built releases are not currently available. See Dockerfile for a step-by-step installation from source.

The general steps are:

  • npm install - install modules needed to build the frontend assets
  • grunt - build the frontend assets
  • pip3 install -r requirements.txt - install python dependencies
  • python3 setup.py install - install Photolib and command line tools

Note that the npm and grunt steps can be skipped if you're not planning to run the web UI (e.g. you want only the command-line tools).

Usage

After installation, run the server:

  • photoappd --port 8080 --library file://./library --database sqlite:///photos.db --cache ./cache

Arguments are as follows:

  • --library file://./library - file storage uri, in this case the relative path ./library
  • --database sqlite:///photos.db - Sqlalchemy connection uri
  • --cache file://./cache - storage uri to use as a cache for things like thumbnails. It can be the same as the library
  • --port 8080 - listen on http on port 8080
  • --thumb-service - optional address of thumbnail service - see below.

Supported library uri schemes are:

  • file - relative (as above) or absolute file paths - file:///srv/library.
  • minio - minio://username:password@host/bucket_name/path/prefix

Photolib is tested with Minio, but should work with S3. For the database, minio is tested with SQLite and MySQL (using the mysql+pymysql:// driver).

Next, the photousers command can be used to create a user account. Login information is necessary to see images marked as private or upload images. You may want to run this before starting the server, but either order works.

  • photousers create -u dave -p mypassword

That's it - you've set up Photolib! Next steps from here would be to the other command line tools to import media.

Besides browsing, most interaction with Photolib will be done using its CLI tool, photocli. Note that server address, username, and password details need to be provided to the cli via a url flag not shown here, or may be stored in a config file - called a context.

Photocli can address multiple photolib instances using contexts. By default, there is one context, named default. After performing the above installation steps, config the CLI to point at your instance:

photocli update-context default --set-host http://<username>:<password>@<host>/
# e.g.:
photocli update-context default --set-host http://myusername:mypassword@localhost:8080/

Optional: Photolib uses a secondary service to generate thumbnails for video files. After the above installation instructions are complete, run photothumbd on a differnet port with the same arguments. On first run, it will create a user for internal communications and log the username/password or you can set THUMBSERVICE_INITIAL_PASSWORD_HASH to the sha256 hash of the desired password. This username/password must be used in the --thumb-service option for photoappd.

Commands

  • photocli ingest - import new photos into the library:
$ photocli ingest photos/*.jpg raws/*.cr2

or add a new format to an existing photo:

$ photocli ingest -c 52679a30-cee4-45c4-8d83-80ee92242e1c photos/*.jpg
  • photocli checkdupes - scan files and print those not already in the library:
$ find source -type f -exec shasum -a 256 {} \; | tee shas.txt
544b5a4f3898abae73260930ee70931a3992c048649692cef89b37576e886f69  source/2018-12-09 13.52.22.jpg
d2c2c30ca4986d364026644881d667162031008e60aac6a69d7b18230b7ea98c  source/2019-04-08 20.05.02.jpg
9d42859ed92d7fb978cf73b41293480e1eab03d2d3a14c185c4daf3a49d324ab  source/2019-06-19 16.28.07.png
...

$ photocli checkdupes --sha-files shas.txt

TODO: support not using --sha-files.

This command, along with ingest, allows for fun tool combinations like so:

photocli checkdupes --sha-files shas.txt | xargs -d "\n" photocli -y ingest

This would ingest all the files listed in shas.txt that aren't already in the database.

Development

Photolib is developed against Python 3.13. Other versions may work, but are not tested.

Env Vars

PHOTOCLI_CONFIG_DIR - override cli config directory location

Roadmap

TODOs

  • Features

    • Big
      • Camera info and camera database
        • show the camera information on each photo page
        • create a catalog of known cameras by extracting the info from the exif
        • associate cameras with photos and allow search by camera
        • cameras can be nicknamed
        • all of this with lenses also?
    • Medium
      • generate several thumbs from videos and present them as a filmstrip - like the 4chan video previews
      • Photo ratings
        • Search by rating
      • Ability to hide photos - hidden option in PhotoStatus enum? Or tag status private?
      • redo feed pager into a <form> so things like pgsize stick
      • Tag management screen
        • Create tag (only available when tagging a photo currently)
        • Count of photos per tag
    • Small
      • "AND" instead of "OR" option for Include/Exclude tag filters
      • improve stats page to differentiate PhotoSets and image files
      • improve stats page to have yearly totals or even a daily view on a calendar
      • portrait/landscape filter in search
      • link on the photo's minimap that takes you to the main map at the location of that photo
      • feature to "download a web safe/optimized copy" - get a compressed jpeg at a common res like 1024x768 with exif stripped
  • Internal

    • need nullable=False on columns where applicable
  • Look and feel

    • give uuids a style like this
    • search next/prev buttons to match feed links
    • search tag selector to match tag picker links
    • <select> styling
    • DARK MODE
    • favicon
    • indicator in thumbnails if something is a video, particularly in search
    • make it clearer that the download/preview button gets a (converted) original size image
    • make the map view markers colored in some logical manner
      • rainbow across age spectrum?
      • 4 colors (with shades) for the 4 seasons?sl
  • Bugs

    • 'preview' link on photo view web pages seem to 500 - not seen recently
    • some video files (that ffmpeg can generate a thumb from) get mime'd as 'audio/mp4' - not seen recently
    • videos under 1s fail to generate a thumb (because of the hard-coded 1s thumb extraction point) - may be fixed, need to review
    • check gps skew in "nearby" section
  • CLI

    • Flesh out
      • data manipulation
        • combine photosets
        • split photosets
      • Relink function - make a photo a member of another photo
      • Config that is saved somewhere
      • Album features
      • Support additional fields on upload like title description tags etc
      • delete features
      • tag features
        • Tag on import
      • modify features (tags & images)
      • "Batch" tag on import
        • Generate a tag on import
        • Save it in config and re-use it (if passing --same-batch)
        • photos imported as a batch will be under 1 tag
      • functions to download images
  • Misc

    • Longer term ideas:

      • "fast ingest" method that touches the db/storage directly. This would scale better than the API ingest.
      • Dynamic svg placeholder for images we can't open
      • Proactive thumbnail generation
      • on photo thumbs on the feed, a little "2" indicating there are 2 images in the set (or w/e number for that item)
      • dark theme
      • more information from the images like on http://exif.regex.info/exif.cgi
      • Map page: use the craigslist style map groups?
    • my photos problems

      • recently imported iom trip photos not time offset
        • time offset options in ui? - should be a search bulk operation

Facial Recognition

Proposed feature. Not yet under development. This section contains notes about its development.

Prior work / modules to use:

The above links are probably all outdated given the recent developments in AI.

Image processing framework

Purpose:

Some way to mass process photos in batches. For example, the batch jobs could: - load metadata from images that is newly supported - process new faces in images - create a downloadable mini-album - proactively generate thumbnails - replace thumbservice, the video thumb service - import photos - also by grabbing them from a foreign source e.g. sftp

Could also have: - a session-based mechanism to "select" photos - e.g. to add them to a batch processing job or album later. Like a clipboard of sorts

Design notes: - general ui: - use a tag multiselector (like the search page) to pick the "input" images for a batch job - for now, we can have a drop-down to select which job to run - there could then be a second-stage config screen for the job where - additional parameters for the job could be set - like what? - the job could run some sort of validation - there is a Job List screen that displays an index of current / past jobs - job executor: - "for now", a SimpleExecutor service. - it's an http service (like thumbservice lol) that is similarly notified by http hook when a job is created by the ui/api (except we pass the ID in the http call this time and it looks up the details from the DB) - internal notes: - tables / cols: - Job: id, uuid, name?, job_name, status(new, running, done, error) - JobTarget: id, job_id, type(tag, set, photo), target - this indicates where a job should get images to work on - there is a numeric ID in the target column that can be any of a Photo, PhotoSet, or Tag - which it is, is determined by the type column - therefore the code can know which method to locate the file with - need testing to ensure the program is usable without having a dedicated executor etc

Description
web-based photo library management software
Readme 1.4 MiB
Languages
Python 68.3%
HTML 19.7%
Less 5.7%
HCL 4.9%
Dockerfile 0.7%
Other 0.7%