User Name
Password
AppleNova Forums » Programmer's Nook »

Building a time lapse script


Register Members List Calendar Search FAQ Posting Guidelines
Building a time lapse script
Thread Tools
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2020-09-22, 13:59

I'm looking to build a time lapse builder that is only using CLI. At this point I wget a URL once per minute via crontab. This the most frequent you can do without any convoluted steps and multiple of the same cron job.

So the command to get the image is:
Code:
wget -q https://camera.ip/snap.jpg -O /var/services/homes/turtle/flexImages/`date +%Y%m%d_%H%M%S`.jpg
This does a "quiet" grab of the image from the URL and saves it to the specified location with the specified name, 20200921_235903.jpg as an example.

This gives me a set of images that can easily be sorted. On 1080p these are between 100k - 350k in size that I've observed so far, but I'm sure that will vary heavily with the camera source.

To do the stitching I use the tried and true ffmpeg. Right now I'm using this, but it is barking at me about deprecated pixel format I haven't searched out yet:
Code:
ffmpeg -r 24 -pattern_type glob -i '*.jpg' -vcodec libx264 -pix_fmt yuv420p /volume1/video/tmp/`date +%Y%m%d`_timelapse.mp4
This goes through the list of files ending in ".jpg" and stitches them together in a 1080p video. By default it is grabbing them in sequence.

The thing I'm working through now is making it so it does one day at a time. I'm thinking I'm going to save 7 days worth of images to be able to build a "mega time lapse" but have one video per day. Right now I can pull through the file list and only grab yesterday's images with "find" but this doesn't work on macOS. It has to do with the difference between BSD and GNU builds for "find" and BSD doesn't support -daystart.
Code:
find /var/services/homes/turtle/flexImages/ -daystart -mtime 1 -ls
Of course, this is a method to list out the images but not really pipe them into ffmpeg. My thought was to move each day's images into their own dated folder (ie, 20200921, 20200922, etc.). Use the above find but with mv instead. Then I get into the run of properly naming a folder for the past etc.

The more I'm thinking about this I think I want to just leave the images in a single folder and have ffmpeg pull them from there. Now I need to figure out how to feed only selected images into ffmpeg like with a for loop or something like that. I also need to make the ffmpeg output silent so it doesn't fill logs with stuff I don't need.

Anyway, the whole point of this is to be able to have cool time lapses of the weather/sky. Mine is going to be for my personal camera, but you can always apply it to something you find at Insecam.

Anyone have any ideas on how to handle just the images from the day before? Since it's going to be scripted I can make a cron executed script that runs after midnight each day. While I could build this as an all encompassing script, I think I'll have it run a a few different for the same of simplicity. One example being the gap command is not even a script but a single command in the cron. I am also running all of this on my NAS since it seems the most compatible for the work and also provides the storage I need.

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2020-09-22, 15:27

I'm not sure I entirely follow, but I wonder if you're looking for a tool like logrotate.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2020-09-22, 15:52

Yeah, I'm not really communicating what I'm looking for well.

The high level overview is I want a daily time lapse video from camera still images. Those videos are built from a single calendar day's worth of images.

The scripting part only gets "complicated" in a few points and those are the ones I'm iffy on of course. Like, how do I get ffmpeg to only look at images from yesterday with the date changing every day. So programmatically, how to I script it to use only the day before's images?

I can list out the day before using find, but how do I translate that into something ffmpeg can use?

I need to spend more time going through ffmpeg source options really. I can always have find spit out a list.txt for the previous day and feed that list the ffmpeg if it can parse the list for image names. I've also used symlinks to handle the folder naming. Like daily have a script create a folder for that day and symlink "today" or "latest" as the folder name. Then I would need to work with Julian dates or something like that though because "day -1" won't work on say 20201001 for folder naming.

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2020-09-22, 16:12

Quote:
Originally Posted by turtle View Post
The scripting part only gets "complicated" in a few points and those are the ones I'm iffy on of course. Like, how do I get ffmpeg to only look at images from yesterday with the date changing every day. So programmatically, how to I script it to use only the day before's images?
You can use `date` as before, but change the relative date:

Code:
echo (date -v-1d +%Y%m%d_%H%M%S)
`-v` means "adjust". `-1d` means "subtract 1 day". You can also do `1d`, which sets the day to 1 (so, September 1st). Or `+1d`, i.e. tomorrow.

Quote:
Originally Posted by turtle View Post
I can list out the day before using find, but how do I translate that into something ffmpeg can use?
You may need xargs here. It lets you take a collection of data, then call something for each item in it.

For example:

Code:
find Downloads -name '*.txt' -print0 | xargs -0 ls -l
This recursively looks through Downloads for names ending with .txt, and calls `ls -l` for each of them. The output will look as though it's all just one big dir, but it's not.

The `-print0` bit here means "don't end these items in a linebreak, but rather with a \0". It's a safer way (if, say, you have some funny characters in your file names) that isn't human-readable, but fine for machine-processing. Then we pipe that to `xargs`, which also gets told to expect \0, using `-0`. And then we tell xargs what to do with each item: call `ls -l`.

It sounds like for this use case, though, you need something more like concatenating multiple files and passing those to ffmpeg? Not sure.

I'm not really sure how you're planning to actually create this timelapse from a series of images. I'm guessing you've already figured that part out.
  quote
drewprops
Bastard
 
Join Date: May 2004
Location: Atlanta
 
2020-09-22, 16:48

I don't know 'nix but I understand the logic. Seems like you're going to have to pick the day before, but nothing from days prior to that, which Chucker seems to be doing?

Didn't realize that FFMPG was scriptable.


...

unnecessary ditty dot loop

...
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2020-09-22, 17:50

I really thought `date -v-1d +%Y%m%d_%H%M%S` was going to save me but it isn't. Turns out that the NAS's Linux doesn't have the "-v" flag in it's build.

That would allow me to do the folders then have the script move and rename folders. I would script the symlink for the "today" folder to it's date, then tell ffmpeg to parse the previous day's folder. That would mean dated folders, but it would also allow me to make it simplified.

Looks like the glob matching that ffmpeg does won't parse a text file but sifts through wildcards. So I would be able to tell it to look in a specific folder and then stitch all the image in that folder together. I can't give it a list to parse that I can tell though. I found a wiki online that talks about the image source and options for it. The options it shows is what I have available.

I'm wondering if I can still kinda make it work by symlinking 20200922 to "today" and then have the script "rename" the symlink to "yesterday" and create a new symlink to "today". Then there is no math for dates but I'm fairly certain symlinks can't be renamed. I can set a variable to the destination folder though allowing me to symlink that....

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.

Last edited by turtle : 2020-09-22 at 18:45.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2020-09-22, 18:41

Well.... this just might work:
Code:
mkdir `date +%Y%m%d` ln -s `date +%Y%m%d` today
Then to get the destination of the symlink...you're gonna love this:
Code:
ls -la|grep today|cut -d"-" -f2|awk -F" " '{print $2}'
I could likely shorten this down a few pipes, but I know the "-" is going to be there so it makes a great point to be the initial delineator. It will not ever change no matter what given it is a symlink. I didn't use the ">" because it doesn't leave me another delineator to work with. the cut leave me with "> 20200922" so I can use the single " " to be the delineator for awk. None of that would be needed if I could just have `date -v-1d`.

So I can use this to set the variable to then remove the symlink and set it to "yesterday" for that date folder.

From there I should be able to run ffmpeg specifying "yesterday" for the folder to be parsed with the glob input:
Code:
ffmpeg -r 24 -pattern_type glob -i '/var/services/homes/turtle/flexImages/yesterday/*.jpg' -vcodec libx264 -pix_fmt yuv420p /volume1/video/tmp/`date +%Y%m%d`_timelapse.mp4
Or something like that anyway. I may have to "cd" to the subfolder and then execute ffmpeg rather than specifying the path. I'll figure that out as I go along on this.

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2020-09-22, 19:02

If you're still looking for some convenient date magic, maybe try some variation of this:

Code:
date --date="yesterday" "+%Y%m%d" date --date="2 day ago" "+%Y%m%d" date --date="3 week ago" "+%Y%m%d"
etc with what you can probably guess for the keywords.

The "--date" argument with a relative "datestr" should work with any moderately recent version of GNU date (not macOS's ancient BSD date).

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2020-09-22, 19:09

That works!
Code:
$ echo `date --date="yesterday" "+%Y%m%d"` 20200921
That will save me from having to do such crazy crap to find the symlink and all.
  quote
kscherer
The Ban Hammer
 
Join Date: Aug 2004
Location: Boyzeee
 
2020-09-23, 11:15

Code:
$=Blah /&%^$&^$ (hyper Noise; generate LOTS)
Just my two cents.



  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2020-09-23, 12:17

That is so insightful! I'm so glad you helped me get that figured out. I couldn't have done it without you.

I'm going to plug away at this again later in the week. I've got a ton on my plate right now but hope to get the scripts worked out for cron jobs by the end of the week. Plus it gives me time to continue collecting images so I have a bunch to work with. Also things like time how long it will take to process the images to the video, etc...

I'll share what I come up with here too, but it might not work on every system since I'm scripting specifically for my NAS and it's version of Linux.

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2020-09-24, 03:24

Quote:
Originally Posted by kscherer View Post
Code:
$=Blah /&%^$&^$ (hyper Noise; generate LOTS)
Just my two cents.



No no no, that's Perl, not bash.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2020-09-25, 14:06

I started looking through the files I've collected so far and a day's worth of snapshots is about 260M. The resulting time-lapse is only 27M. This is very doable to even save more than 7 days worth.

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2020-09-25, 19:59

Code:
#!/bin/bash # # Copy daily images into folder for previous day, then compile into time lapse. # log=/var/services/homes/turtle/dailySnapshotCron.log yesterday=`date --date="yesterday" "+%Y%m%d"` path=/var/services/homes/turtle/flexImages/ # Start Logging event echo `date +%Y%m%d_%H%M%S` Begun daily actions >> $log # Make dir for yesterday and copy images to it mkdir $path$yesterday find $path -daystart -mtime 1 -size +20k -exec cp -t $path$yesterday {} + # Change to yesterday and nicely run time lapse compilation using ffmpeg cd $path$yesterday nice ffmpeg -hide_banner -loglevel warning -r 24 -pattern_type glob -i '*.jpg' -vcodec libx264 -pix_fmt yuv420p '/volume1/video/tmp/'$yesterday'_timelapse.mp4' >> $log # Remove "yesterday" dir rm -rf $path$yesterday # Closing log echo `date +%Y%m%d_%H%M%S` Completed daily actions >> $log # Unset variables unset log; unset yesterday; unset path
That is the code that will run daily via cron and produce the time lapse videos. Each video is about 30M as I mentioned before. This will take a long time, if I forget to purge, to fill the NAS. I'm copying the images to the folder so I still have one folder housing all the snapshots in a general sense. This will be helpful when I do the weekly time lapse video.

To breakdown the script for those who don't follow:
  • Set variables
    I went with variables like this to allow for flexibility for others using this script. Basically, use it and update your paths as needed.
  • Basic log for the start
    I wanted to log the start and end of the run so I had an idea of how long. It isn't needed, but it takes about 8 minutes on my NAS to run and it is fairly underpowered.
  • Make a directory to send the day's snapshots too
    This is a fun part. It makes the directory where you are going to send the day's snapshots too and then copies those snapshots over. I had to add a filter to the find command because the bad snapshot grabs resulted in 0b files and that stopped ffmpeg in its tracks. Putting the filter keeps it with assumably valid image files. Adjust that size as needed if you are pulling smaller files.
  • Change to that dir and compile the snapshots using nice
    ffmpeg doesn't like pulling globs from a path so you change to that dir. I run it with "nice" because it is a very underpowered NAS and it keeps everything else running mildly well while the process is running. nice can be adjusted as needed, defaults work fine for this.
  • Clean up
    I'm removing the "yesterday" directory because I don't want it hanging around. Since I'm using copy early on, I still have the originals in the source dir.
  • Close log entry
    Again, not required, but helps me know how long it too to run.
  • unset variables
    Old habit though not technically needed, I unset my variables even though they don't live beyond the script.

I still have the barking about deprecated pixel format... but I don't really care so I'm not in a rush to change it. Apparently it has something to do with the "-pix_fmt yuv4420p" portion.

The resulting log looks like this:
Code:
20200925_201851 Begun daily actions 20200925_202634 Completed daily actions
The video is named: 20200924_timelapse.mp4

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
drewprops
Bastard
 
Join Date: May 2004
Location: Atlanta
 
2020-09-25, 22:42

Cool summation.


...
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2020-09-29, 08:33

The script is running great automated. Currently I have images from the 22nd until now. It is at almost 10600 images and only 2.1GB in size for all those images.

Of the 5 time lapse videos that have been created the largest is 37.8MB. Most are only 31MB in size.

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2020-10-26, 13:27

I forgot about this and haven't automated the purging of old files so I figured I would post where I'm at now.

49,784 images for a total of only 9.7GB.

There have been 36 videos made totaling only 929MB!

Guess it isn't really a problem that I've forgotten to script in pruning.

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
Posting Rules Navigation
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Post Reply

Forum Jump
Thread Tools
Similar Threads
Thread Thread Starter Forum Replies Last Post
Building a PC Kraetos General Discussion 103 2008-02-15 13:40
Building a PVR alcimedes Third-Party Products 11 2007-08-30 16:28
Building PHP 5.1 rollercoaster375 Genius Bar 4 2005-12-15 05:44
Building a PC faramirtook Purchasing Advice 7 2005-06-23 17:18
Building a PC....that's right. Akumulator General Discussion 20 2004-11-10 11:26


« Previous Thread | Next Thread »

All times are GMT -5. The time now is 03:37.


Powered by vBulletin®
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright ©2004 - 2020, AppleNova