2019-07-04 17:29:49 -07:00
|
|
|
Photolib
|
2019-06-21 09:22:42 -07:00
|
|
|
========
|
|
|
|
|
|
|
|
Cloud-native photo and related media library software, with a focus for photography hobbyists.
|
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
Photolib provides a simple, rich, and modern UI to manage your photo and video library. It can scale to host hundreds of
|
|
|
|
thousands of photos but is also light and simple enough to host just the couple photos you shoot with your smartphone.
|
|
|
|
|
|
|
|
Photolib stores images as a logical "photoset". This allows identical images and metadata to be grouped into one item.
|
|
|
|
Many photographers shoot in multiple formats - that is, to produce multiple files per click of the shutter. A JPEG and
|
|
|
|
a RAW of some sort (Like Canon's CR2) is common. The primary idea behind Photolib's organization is that one click of a
|
|
|
|
camera shutter equals one photoset.
|
|
|
|
|
|
|
|
When photos are imported - assuming they are done in the same batch - files are grouped based on their name and
|
|
|
|
extension, using rules that apply to almost every camera format. For example, a camera shooting under the "multiple
|
|
|
|
formats" mode described above would create, from one shot, `IMG_1234.JPG` and `IMG_1234.CR2`. Photolib would detect that
|
|
|
|
since these two images have the same file name (the `IMG_1234` part), they are the same photo and should be grouped into
|
|
|
|
a photoset.
|
|
|
|
|
|
|
|
In addition to controls in the web interface, Photolib provides both a REST API and command-line tools to manage your
|
|
|
|
photo library.
|
2019-06-21 09:22:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
Installation
|
|
|
|
------------
|
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
Pre-built releases are not currently available. See `Dockerfile` for a step-by-step installation from source.
|
|
|
|
|
|
|
|
The general steps are:
|
|
|
|
|
|
|
|
* `npm install` - install modules needed to build the frontend assets
|
|
|
|
* `grunt` - build the frontend assets
|
|
|
|
* `pip3 install -r requirements.txt` - install python dependencies
|
|
|
|
* `python3 setup.py install` - install Photolib and command line tools
|
|
|
|
|
|
|
|
Note that the npm and grunt steps can be skipped if you're not planning to run the web UI (e.g. you want only the
|
|
|
|
command-line tools).
|
2019-06-21 09:22:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
Usage
|
|
|
|
-----
|
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
After installation, run the server:
|
2019-06-21 09:22:42 -07:00
|
|
|
|
2019-07-06 11:25:15 -07:00
|
|
|
* `photoappd --port 8080 --library file://./library --database sqlite:///photos.db --cache ./cache`
|
2019-06-21 09:22:42 -07:00
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
Arguments are as follows:
|
2019-06-21 09:22:42 -07:00
|
|
|
|
2019-07-06 11:25:15 -07:00
|
|
|
* `--library file://./library` - file storage uri, in this case the relative path `./library`
|
|
|
|
* `--database sqlite:///photos.db` - [Sqlalchemy](https://docs.sqlalchemy.org/en/13/core/engines.html) connection uri
|
2019-07-11 19:26:22 -07:00
|
|
|
* `--cache file://./cache` - storage uri to use as a cache for things like thumbnails. It can be the same as the library
|
2019-07-04 17:29:49 -07:00
|
|
|
* `--port 8080` - listen on http on port 8080
|
2021-08-22 15:47:15 -07:00
|
|
|
* `--thumb-service` - optional address of thumbnail service - see below.
|
2019-06-21 09:22:42 -07:00
|
|
|
|
2019-07-06 11:25:15 -07:00
|
|
|
Supported library uri schemes are:
|
|
|
|
|
|
|
|
* file - relative (as above) or absolute file paths - `file:///srv/library`.
|
|
|
|
* minio - `minio://username:password@host/bucket_name/path/prefix`
|
|
|
|
|
|
|
|
Photolib is tested with [Minio](https://min.io/), but should work with S3. For the database, minio is tested with SQLite
|
|
|
|
and MySQL (using the `mysql+pymysql://` driver).
|
|
|
|
|
2019-07-04 18:41:57 -07:00
|
|
|
Next, the `photousers` command can be used to create a user account. Login information is necessary to see images marked
|
2019-07-06 11:25:15 -07:00
|
|
|
as private or upload images. You may want to run this before starting the server, but either order works.
|
2019-06-21 09:22:42 -07:00
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
* `photousers create -u dave -p mypassword`
|
|
|
|
|
|
|
|
That's it - you've set up Photolib! Next steps from here would be to the other command line tools to import media.
|
2019-06-21 09:22:42 -07:00
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
Besides browsing, most interaction with Photolib will be done using its CLI tool, `photocli`. Note that server address,
|
2019-07-16 21:15:49 -07:00
|
|
|
username, and password details need to be provided to the cli via a url flag not shown here. See `photocli --help` for
|
|
|
|
more information.
|
2019-06-21 09:22:42 -07:00
|
|
|
|
2021-08-22 15:47:15 -07:00
|
|
|
Optional: Photolib uses a secondary service to generate thumbnails for video files. After the above installation
|
|
|
|
instructions are complete, run `photothumbd` on a differnet port with the same arguments. On first run, it will create
|
2021-08-25 23:21:11 -07:00
|
|
|
a user for internal communications and log the username/password or you can set `THUMBSERVICE_INITIAL_PASSWORD_HASH` to
|
|
|
|
the sha256 hash of the desired password. This username/password must be used in the `--thumb-service` option
|
|
|
|
for `photoappd`.
|
2021-08-22 15:47:15 -07:00
|
|
|
|
2019-06-21 09:22:42 -07:00
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
Commands
|
|
|
|
--------
|
|
|
|
|
2019-06-21 09:22:42 -07:00
|
|
|
* `photocli ingest` - import new photos into the library:
|
|
|
|
|
|
|
|
|
|
|
|
```
|
2019-07-04 17:29:49 -07:00
|
|
|
$ photocli ingest photos/*.jpg raws/*.cr2
|
|
|
|
```
|
|
|
|
|
|
|
|
or add a new format to an existing photo:
|
|
|
|
|
|
|
|
```
|
|
|
|
$ photocli ingest -c 52679a30-cee4-45c4-8d83-80ee92242e1c photos/*.jpg
|
|
|
|
```
|
|
|
|
|
2019-06-21 09:22:42 -07:00
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
* `photocli checkdupes` - scan files and print those not already in the library:
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
$ find source -type f -exec shasum -a 256 {} \; | tee shas.txt
|
|
|
|
544b5a4f3898abae73260930ee70931a3992c048649692cef89b37576e886f69 source/2018-12-09 13.52.22.jpg
|
|
|
|
d2c2c30ca4986d364026644881d667162031008e60aac6a69d7b18230b7ea98c source/2019-04-08 20.05.02.jpg
|
|
|
|
9d42859ed92d7fb978cf73b41293480e1eab03d2d3a14c185c4daf3a49d324ab source/2019-06-19 16.28.07.png
|
2019-06-21 09:22:42 -07:00
|
|
|
...
|
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
$ photocli checkdupes --sha-files shas.txt
|
2019-06-21 09:22:42 -07:00
|
|
|
```
|
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
TODO: support not using `--sha-files`.
|
|
|
|
|
|
|
|
This command, along with `ingest`, allows for fun tool combinations like so:
|
|
|
|
|
|
|
|
```
|
|
|
|
photocli checkdupes --sha-files shas.txt | xargs -d "\n" photocli -y ingest
|
|
|
|
```
|
2019-06-21 09:22:42 -07:00
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
This would ingest all the files listed in `shas.txt` that aren't already in the database.
|
|
|
|
|
|
|
|
|
|
|
|
Roadmap
|
|
|
|
-------
|
2022-12-22 10:58:11 -08:00
|
|
|
|
|
|
|
- TODOs:
|
2023-01-12 22:59:19 -08:00
|
|
|
- need `nullable=False` on columns where applicable
|
|
|
|
- 0-5 rating option for photosets
|
2022-12-22 10:58:11 -08:00
|
|
|
- give uuids a style like `this`
|
|
|
|
- search next/prev buttons to match feed links
|
|
|
|
- search tag selector to match tag picker links
|
|
|
|
- <select> styling
|
|
|
|
- improve stats page to differentiate PhotoSets and image files
|
|
|
|
- improve stats page to have yearly totals or even a daily view on a calendar
|
|
|
|
- redo feed pager into a <form> so things like `pgsize` stick
|
|
|
|
- portrait/landscape filter in search
|
|
|
|
- DARK MODE
|
|
|
|
- favicon
|
|
|
|
- indicator in thumbnails if something is a video, particularly in search
|
|
|
|
- search by type of image or video
|
|
|
|
- 'preview' link on photo view web pages seem to 500
|
|
|
|
- some video files (that ffmpeg can generate a thumb from) get mime'd as 'audio/mp4'
|
|
|
|
- videos under 1s fail to generate a thumb (because of the hard-coded 1s thumb extraction point)
|
|
|
|
- link on the photo's minimap that takes you to the main map at the location of that photo
|
|
|
|
- nullable=False for all columns possible
|
|
|
|
- check/clean no-js fallbacks
|
|
|
|
- check gps skew in "nearby" section
|
2023-01-12 22:59:19 -08:00
|
|
|
- camera info / database
|
|
|
|
- show the camera information on each photo page
|
|
|
|
- create a catalog of known cameras by extracting the info from the exif
|
|
|
|
- associate cameras with photos and allow search by camera
|
|
|
|
- cameras can be nicknamed
|
|
|
|
- all of this with lenses also?
|
|
|
|
- make it clearer that the download/preview button gets a (converted) original size image
|
|
|
|
- make the map view markers colored in some logical manner
|
|
|
|
- rainbow across age spectrum?
|
|
|
|
- 4 colors (with shades) for the 4 seasons?
|
|
|
|
- colorblindness?
|
2022-12-22 10:58:11 -08:00
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
- Flesh out CLI:
|
2021-07-07 19:50:15 -07:00
|
|
|
- data manipulation
|
|
|
|
- combine photosets
|
|
|
|
- split photosets
|
2019-07-11 09:16:14 -07:00
|
|
|
- Relink function - make a photo a member of another photo
|
2019-07-04 17:29:49 -07:00
|
|
|
- Config that is saved somewhere
|
2019-07-11 09:16:14 -07:00
|
|
|
- Album features
|
2019-07-04 17:29:49 -07:00
|
|
|
- Support additional fields on upload like title description tags etc
|
|
|
|
- delete features
|
|
|
|
- tag features
|
2019-07-11 09:16:14 -07:00
|
|
|
- Tag on import
|
2019-07-04 17:29:49 -07:00
|
|
|
- modify features (tags & images)
|
2019-07-11 09:16:14 -07:00
|
|
|
- "Batch" tag on import
|
|
|
|
- Generate a tag on import
|
|
|
|
- Save it in config and re-use it (if passing --same-batch)
|
|
|
|
- photos imported as a batch will be under 1 tag
|
2022-12-22 10:58:11 -08:00
|
|
|
|
2019-07-04 17:29:49 -07:00
|
|
|
- Longer term ideas:
|
|
|
|
- "fast ingest" method that touches the db/storage directly. This would scale better than the API ingest.
|
2019-07-04 18:41:57 -07:00
|
|
|
- Dynamic svg placeholder for images we can't open
|
2019-07-06 13:49:16 -07:00
|
|
|
- Proactive thumbnail generation
|
2019-07-13 15:47:59 -07:00
|
|
|
- on photo thumbs on the feed, a little "2" indicating there are 2 images in the set (or w/e number for that item)
|
|
|
|
- dark theme
|
|
|
|
- more information from the images like on http://exif.regex.info/exif.cgi
|
2022-12-22 10:58:11 -08:00
|
|
|
- Map page: use the craigslist style map groups?
|
2019-07-13 15:47:59 -07:00
|
|
|
|
2021-07-07 19:50:15 -07:00
|
|
|
- my photos problems
|
|
|
|
- recently imported imotrip photos not time offset lol
|
|
|
|
- time offset options in ui?
|
2022-12-22 10:58:11 -08:00
|
|
|
|
|
|
|
- Low-priority TODOs
|
|
|
|
- Cli functions to download images
|
|
|
|
- feature to "download a web safe/optimized copy" - get a compressed jpeg at a common res like 1024x768 with exif stripped
|
|
|
|
|
|
|
|
|
|
|
|
Facial Recognition
|
|
|
|
^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
Proposed feature. Not yet under development. This section contains notes about its development.
|
|
|
|
|
|
|
|
Prior work / modules to use:
|
|
|
|
|
|
|
|
* https://github.com/ageitgey/face_recognition/blob/master/examples/find_faces_in_picture.py
|
|
|
|
* https://face-recognition.readthedocs.io/en/latest/face_recognition.html
|
|
|
|
* https://www.pyimagesearch.com/2018/06/18/face-recognition-with-opencv-python-and-deep-learning/
|
|
|
|
* https://github.com/JohannesBuchner/imagehash
|
2023-01-12 22:59:19 -08:00
|
|
|
|
|
|
|
|
|
|
|
Image processing framework
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
Purpose:
|
|
|
|
|
|
|
|
Some way to mass process photos in batches. For example, the batch jobs could:
|
|
|
|
- load metadata from images that is newly supported
|
|
|
|
- process new faces in images
|
|
|
|
- create a downloadable mini-album
|
|
|
|
- proactively generate thumbnails
|
|
|
|
- replace thumbservice, the video thumb service
|
|
|
|
|
|
|
|
Could also have:
|
|
|
|
- a session-based mechanism to "select" photos
|
|
|
|
- e.g. to add them to a batch processing job or album later. Like a clipboard of sorts
|
|
|
|
|
|
|
|
Design notes:
|
|
|
|
- general ui:
|
|
|
|
- use a tag multiselector (like the search page) to pick the "input" images for a batch job
|
|
|
|
- for now, we can have a drop-down to select which job to run
|
|
|
|
- there could then be a second-stage config screen for the job where
|
|
|
|
- additional parameters for the job could be set
|
|
|
|
- like what?
|
|
|
|
- the job could run some sort of validation
|
|
|
|
- there is a Job List screen that displays an index of current / past jobs
|
|
|
|
|
|
|
|
- job executor:
|
|
|
|
- "for now", a SimpleExecutor service.
|
|
|
|
- it's an http service (like thumbservice lol) that is similarly notified by http hook when a job is created by
|
|
|
|
the ui/api (except we pass the ID in the http call this time and it looks up the details from the DB)
|
|
|
|
|
|
|
|
- internal notes:
|
|
|
|
- tables / cols:
|
|
|
|
- Job: id, uuid, name?, job_name, status(new, running, done, error)
|
|
|
|
- JobTarget: id, job_id, type(tag, set, photo), target
|
|
|
|
- this indicates where a job should get images to work on
|
|
|
|
- there is a numeric ID in the `target` column that can be any of a Photo, PhotoSet, or Tag
|
|
|
|
- which it is, is determined by the `type` column
|
|
|
|
- therefore the code can know which method to locate the file with
|