User Name
Password
AppleNova Forums » Programmer's Nook »

Anyone one here good with Python? (Daemonize)


Register Members List Calendar Search FAQ Posting Guidelines
Anyone one here good with Python? (Daemonize)
Thread Tools
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2021-05-28, 17:18

I need to daemonize a script rather than run it in a screen session. It works now, but once it crashes or whatever I need it to run again without me having to realize it stopped.

The background is I'm using a Raspberry Pi to serve as a temperature and humidity data logger. The original guide I followed use an older Adafruit_Python_DHT that has been deprecated. The newer script version is called CircuitPython-DHT. I've actually modified my copy so it doesn't send the alerts right to my Zabbix server but rather writes the temp and humidity readings to a txt file so it can be pulled by Zabbix.

This is my current script with my modifications:
Code:
# SPDX-FileCopyrightText: 2021 ladyada for Adafruit Industries # SPDX-License-Identifier: MIT import time import board import adafruit_dht # Initial the dht device, with data pin connected to: #dhtDevice = adafruit_dht.DHT22(board.D2) # you can pass DHT22 use_pulseio=False if you wouldn't like to use pulseio. # This may be necessary on a Linux single board computer like the Raspberry Pi, # but it will not work in CircuitPython. dhtDevice = adafruit_dht.DHT22(board.D2, use_pulseio=False) while True: try: # Print the values to the serial port temperature_c = dhtDevice.temperature temperature_f = temperature_c * (9 / 5) + 32 humidity = dhtDevice.humidity print( "Temp: {:.1f} F / {:.1f} C Humidity: {}% ".format( temperature_f, temperature_c, humidity ) ) with open('/tmp/temp', 'w') as f: f.write("{:.1f}".format( temperature_f ) ) with open('/tmp/humidity', 'w') as f: f.write("{}".format( humidity ) ) except RuntimeError as error: # Errors happen fairly often, DHT's are hard to read, just keep going print(error.args[0]) time.sleep(2.0) continue except Exception as error: dhtDevice.exit() raise error time.sleep(5.0)
I know I can remove the "print" but I'm using that since I'm running it in screen. If I can demonize it that would go away. I looked at this article that talks about how to daemonize a script but I can't get it to work like I'm expecting and I know it is from my lack of knowledge of Python. Heck, it barked at me because I mixed spaces and tabs to indentation.

This is what I came up with but it doesn't do anything and no news is not good news in this case. There is no process running and I would expect nor a PID file ever actually created:
Code:
#!/usr/bin/python3 # SPDX-FileCopyrightText: 2021 ladyada for Adafruit Industries # SPDX-License-Identifier: MIT from daemonize import Daemonize import time import board import adafruit_dht pid = "/tmp/dataDHT.pid" # Initial the dht device, with data pin connected to: dhtDevice = adafruit_dht.DHT22(board.D2) def main(): while True: try: # Print the values to the serial port temperature_c = dhtDevice.temperature temperature_f = temperature_c * (9 / 5) + 32 humidity = dhtDevice.humidity #print( # "Temp: {:.1f} F / {:.1f} C Humidity: {}% ".format( # temperature_f, temperature_c, humidity # ) #) with open('/tmp/temp', 'w') as f: f.write("{:.1f}".format( temperature_f ) ) with open('/tmp/humidity', 'w') as f: f.write("{}".format( humidity ) ) except RuntimeError as error: # Errors happen fairly often, DHT's are hard to read, just keep going print(error.args[0]) time.sleep(2.0) continue except Exception as error: dhtDevice.exit() raise error time.sleep(5.0) daemon = Daemonize(app="dataDHT", pid=pid, action=main) daemon.start()

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2021-05-28, 17:33

So this needs to run every roughly 10 seconds?

Consider getting rid of the loop altogether and instead making a systemd unit that calls the script at an interval. Something like:

Code:
[Unit] Description=Read DHT sensor [Service] User=%I Restart=always RestartSec=10s ExecStart=#!/usr/bin/python3 /path/to/your/script.py
Put that in /etc/systemd/system/read-dht-sensor.service and then do systemctl enable read-dht-sensor --now (I think).

I'm assuming this is on Raspbian and/or that you have systemd.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2021-05-29, 13:44

Ohhh...

I didn't even think of something like that. I like it! Now I just need to figure out how to get rid of the loop. I'll get it, I just don't know python well at all. Kinda new territory to me. I'll post back for help with it if I can't get it figured out... or even if I do with what my code looks like.

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2021-05-29, 15:01

Well, that seems to work but the errors are a problem. DHT22 is known for having errors on readings so this isn't a shock. I need error handling for this script now. This is my result when manually running it a number of times:
Code:
pi@pitemp:~ $ ./dataDHT.py Traceback (most recent call last): File "./dataDHT.py", line 13, in <module> temperature_c = dhtDevice.temperature File "/home/pi/.local/lib/python3.7/site-packages/adafruit_dht.py", line 259, in temperature self.measure() File "/home/pi/.local/lib/python3.7/site-packages/adafruit_dht.py", line 215, in measure raise RuntimeError("A full buffer was not returned. Try again.") RuntimeError: A full buffer was not returned. Try again. pi@pitemp:~ $ ./dataDHT.py Traceback (most recent call last): File "./dataDHT.py", line 13, in <module> temperature_c = dhtDevice.temperature File "/home/pi/.local/lib/python3.7/site-packages/adafruit_dht.py", line 259, in temperature self.measure() File "/home/pi/.local/lib/python3.7/site-packages/adafruit_dht.py", line 215, in measure raise RuntimeError("A full buffer was not returned. Try again.") RuntimeError: A full buffer was not returned. Try again. pi@pitemp:~ $ ./dataDHT.py Temp: 74.5 F / 23.6 C Humidity: 53.5%
This is the code now, minus some of the commented sections for easy reading:
Code:
#!/usr/bin/python3 import time import board import adafruit_dht # Initial the dht device, with data pin connected to: dhtDevice = adafruit_dht.DHT22(board.D2) # Print the values to the serial port temperature_c = dhtDevice.temperature temperature_f = temperature_c * (9 / 5) + 32 humidity = dhtDevice.humidity print( "Temp: {:.1f} F / {:.1f} C Humidity: {}% ".format( temperature_f, temperature_c, humidity ) ) with open('/tmp/temp', 'w') as f: f.write("{:.1f}".format( temperature_f ) ) with open('/tmp/humidity', 'w') as f: f.write("{}".format( humidity ) )
I enabled the "print" section so I could see in the terminal what was happening rather than just checking the text files.

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2021-05-29, 15:09

Yeah, I would leave the try/except stuff in.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2021-05-29, 15:23

That seems to be helping. I didn't know the format for loops in python so I just stripped everything out based n indention. I put them back and so far it seems to be working. I'm still tweaking it and such so I'll post my end result after I the point I think I'll settle with.

I'm testing right now in screen with this:
Code:
watch -n5 ./dataDHT.py
It is actually working too.

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2021-05-29, 15:41

Yeah, Python is all about indentation — the while ends wherever the indentation increases. (I get the appeal, but… not a fan.)
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2021-05-29, 16:11

Looks like you got this solved before I saw it, but I guess I'll add my two cents on a few small things…

Quote:
Originally Posted by turtle View Post
I know I can remove the "print" but I'm using that since I'm running it in screen.
Although using plain "print" is (rightfully) shunned for libraries and some backend services, I think it's perfectly fine for a one-off personal script like this, especially since you could simply redirect stdout/stderr into a couple of files and then look at those files in an independent shell or process to observe your script's behavior.

Quote:
Originally Posted by turtle View Post
Heck, it barked at me because I mixed spaces and tabs to indentation.
Welcome to Python. "Whitespace is important", as some Pythonistas would say. I'm sure it feels weird at first, but most folks I've known come to love Python's formatting rules pretty quickly, at least if they're writing a lot of Python on a daily basis. If you're interested in doing a lot more Python coding, I'd be happy to talk at length about PEP 8, flake8, and black for more opinionated general code formatting.

Quote:
Originally Posted by turtle View Post
There is no process running and I would expect nor a PID file ever actually created:
It may not be obvious at a glance, but I think this script is using the third-party daemonize library. If you didn't install that yourself, I wouldn't be surprised if it was already installed by some other Raspberry Pi software. I think I tried using daemonize many years ago, and I vaguely remember it being pretty fiddly, and I think I gave up on it. In general, using this (or the better-supported python-daemon library) likely requires a bit deeper Python knowledge than you have right now. You probably want to be familiar with context managers, threads, signal handling, and various file handle objects to make this work well; you might be better off just letting the caller deal with the process/stdin/stdout/signals management.

I like chucker's systemd service suggestion and would recommend that over manual daemon setup.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2021-05-29, 16:28

Quote:
Originally Posted by Brad View Post
...
Welcome to Python. "Whitespace is important", as some Pythonistas would say. I'm sure it feels weird at first, but most folks I've known come to love Python's formatting rules pretty quickly, at least if they're writing a lot of Python on a daily basis. If you're interested in doing a lot more Python coding, I'd be happy to talk at length about PEP 8, flake8, and black for more opinionated general code formatting.
So relevant here.

I mean, I remember watching these episodes and then hearing about it from my boss who liked it one way over the other. (I completely forgot what it was since I'm not a developer and didn't care and now he's quit our company and is no longer our boss. )

Quote:
It may not be obvious at a glance, but I think this script is using the third-party daemonize library. If you didn't install that yourself, I wouldn't be surprised if it was already installed by some other Raspberry Pi software. I think I tried using daemonize many years ago, and I vaguely remember it being pretty fiddly, and I think I gave up on it. In general, using this (or the better-supported python-daemon library) likely requires a bit deeper Python knowledge than you have right now. You probably want to be familiar with context managers, threads, signal handling, and various file handle objects to make this work well; you might be better off just letting the caller deal with the process/stdin/stdout/signals management.

I like chucker's systemd service suggestion and would recommend that over manual daemon setup.
I'm certainly looking to get better with Python but not really trying to make it a day Job thing for me. I like a good general knowledge enough to be able to modify someone else's code for my personal uses. This means I don't have to learn everything and can typically get the results I'm after. I'm going to skip the demonizing thing and go with chucker's plan since it seems to fit the best for what I'm doing. I just thought it would be a "plug and play" kinda thing and it was far from it.

In fact, this is how most of my "coding" gets done that isn't bash scripting. I actually am pretty good with bash so I have that going for me.

Louis L'Amour, “To make democracy work, we must be a notion of participants, not simply observers. One who does not vote has no right to complain.”
MineCraft? mc.applenova.com | Visit us! | Maybe someday I'll proof read, until then deal with it.
  quote
tomoe
Veteran Member
 
Join Date: Nov 2006
Send a message via AIM to tomoe  
2021-05-29, 18:04

❤️ black & flake8 checking. Have both of those setup as pre-commit hooks on my work laptop and as part of the CI/CD.

The logging module can be pretty handy if you want to move beyond print statements, e.g.

Code:
import logging LOGGER = logging.getLogger(__name__) while True: try: # Log variables, return values, etc. LOGGER.info("Some variable in the code is: {my_variable}".format(my_variable=my_variable)) except RuntimeError as err: LOGGER.error("Whoops some error occurred", exc_info=True) LOGGER.info(f"Here is some other variable: {some_other}".format(some_other=some_other))

Seen a man standin' over a dead dog lyin' by the highway in a ditch
He's lookin' down kinda puzzled pokin' that dog with a stick
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2021-05-29, 19:31

Quote:
Originally Posted by tomoe View Post
❤️ black & flake8 checking. Have both of those setup as pre-commit hooks on my work laptop and as part of the CI/CD.
Nice! We call flake8 (with flake8-black) from a tox env we run in CI and/or locally. We don't use commit hooks, but to protect the repo, we simply forbid anyone from committing or pushing directly to master, and all code must come through pull requests with passing flake8/black/coverage/etc. jobs before they can merge. The green checks are a beautiful thing!



I actually have my local editor set up to automatically run black every time it saves a Python file. I resisted auto-formatters at first, but getting to just turn off that part of the brain and never think about debating style choices is soooooooo good.

Quote:
Originally Posted by tomoe View Post
The logging module can be pretty handy if you want to move beyond print statements, e.g.
100%. The logging module is definitely an upgrade for generating good console output, especially for any long-running code. What's also great about using the logging module (this is probably more for turtle's education…) is that you can configure the logging module at a high level with different handlers and formatters to, for example, send some logs to files on disk and some logs to stdout, or prepend the timestamp and process ID to all messages, or only output things logged at "error" in one part of your code but output everything at "debug" in some other part of your code. Heck, some of our Python services have a logging handler that intercepts log messages and sends them across the internet into AWS CloudWatch storage! Python logging configuration is very powerful, though understanding all the options can be overwhelming at first.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2021-05-29, 20:02

Quote:
Originally Posted by turtle View Post
Kind of sort of. That's pretty cringey to watch, but yeah I imagine there really are some folks out there who are exactly that awkward.

For the most part, Python doesn't care whether you use tabs or spaces, but it does need for you to pick a lane. A given file should be indented exclusively by one or the other, never mixing both, because one programmer's definition of a tabstop width may be completely different from the next programmer's (1 tab = 4 spaces? 8? 2???), and Python doesn't want to pick sides in that argument. Instead, it basically throws up its arms and makes you correct the inconsistency.

Most editors and IDEs, even vim in the terminal, can be taught to insert the right amount of space characters when you hit the tab key. I'm pretty sure 99% of my indended spaces in recent years originated from me hitting the tab key and trusting my editor to do the right thing.

("The right thing" is "convert to spaces", obviously. )

Quote:
Originally Posted by turtle View Post
I'm certainly looking to get better with Python but not really trying to make it a day Job thing for me. I like a good general knowledge enough to be able to modify someone else's code for my personal uses. This means I don't have to learn everything and can typically get the results I'm after. […] In fact, this is how most of my "coding" gets done that isn't bash scripting. I actually am pretty good with bash so I have that going for me.
No better time than the present to learn more Python! I've been using Python professionally for about ten years now across different kinds of projects, but it's also been my go-to tool for lots of one-off scripts and simple projects at home. Usually my thinking goes: Can I solve this by piping a bunch of commands together like grep | cut | sort | uniq? If not, then can I solve it using a shell script with simple variables and loops and conditions? If not, Python!

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2021-05-30, 03:27

Quote:
Originally Posted by Brad View Post
I've been using Python professionally for about ten years now across different kinds of projects
I'm so sorry.
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2021-05-30, 10:34

Quote:
Originally Posted by chucker View Post
I'm so sorry.
Pretty much all the worthwhile options around me in recent years have been Java (or Scala), Ruby, or Python, and I'll take Python from that list any day of the week. I've seen a small smattering of Node stuff and a light dusting of Go, but those have been few and far between.

Though, for a brief while in 2017 one company desperately tried to get me to write .NET 2.0 (released 2005) running on Windows Server 2008. I've not seen many dumpster fires quite as gnarly as that one.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2021-05-30, 11:26

Quote:
Originally Posted by Brad View Post
Pretty much all the worthwhile options around me in recent years have been Java (or Scala), Ruby, or Python, and I'll take Python from that list any day of the week. I've seen a small smattering of Node stuff and a light dusting of Go, but those have been few and far between.

Though, for a brief while in 2017 one company desperately tried to get me to write .NET 2.0 (released 2005) running on Windows Server 2008. I've not seen many dumpster fires quite as gnarly as that one.
Heh. I do still occasionally have a client call me whose stack runs on .NET 2.0 for… reasons. (Budget. It's always budget.)

I used to write some Ruby back in the 2000s, and I think it has some cute features, but I get the sense I could never go back to static typing.
  quote
tomoe
Veteran Member
 
Join Date: Nov 2006
Send a message via AIM to tomoe  
2021-05-30, 12:46

Quote:
Originally Posted by Brad View Post
Nice! We call flake8 (with flake8-black) from a tox env we run in CI and/or locally. We don't use commit hooks, but to protect the repo, we simply forbid anyone from committing or pushing directly to master, and all code must come through pull requests with passing flake8/black/coverage/etc. jobs before they can merge. The green checks are a beautiful thing!
Yes! I used tox initially, but eventually stopped using it as we migrated/started using Spark a lot more. Tox doesn't (or at least didn't) play well with spark at all. We do product the main, staging, and prod branches such that all status checks must pass before a commit can be merged. We used coverage for awhile, but ended up cancelling the service we were using (codecov), so have since stopped using those checks (to my annoyance, but I'm in data science/engineering so don't have a ton of sway there.) I did love having the option to set minimum coverage thresholds for new code in pull requests, requirements that coverage can'd decrease more than X%, etc, etc.

Quote:
Originally Posted by Brad View Post
I resisted auto-formatters at first, but getting to just turn off that part of the brain and never think about debating style choices is soooooooo good.
This has been one of the best parts for us as well. There was *one* time where black couldn't reformat a line such that it stayed with 100 characters. However, on inspection in the pull request, I noticed the engineer had tried to commit code that had this insanely nested lambdas of lambdas of lambdas (aka something that was going to be a nightmare to debug & maintain). That...got fixed.

Seen a man standin' over a dead dog lyin' by the highway in a ditch
He's lookin' down kinda puzzled pokin' that dog with a stick
  quote
Posting Rules Navigation
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Post Reply

Forum Jump
Thread Tools
Similar Threads
Thread Thread Starter Forum Replies Last Post
New to Python, pointers? turtle Programmer's Nook 22 2014-11-21 20:45
fast python input evan Programmer's Nook 12 2010-10-28 15:22
Need some massive help in Python... RowdyScot Programmer's Nook 4 2006-10-10 00:51
Python in Xcode Oskar Programmer's Nook 7 2006-02-04 16:23
Python vs Alligator Matsu AppleOutsider 3 2005-10-06 22:10


« Previous Thread | Next Thread »

All times are GMT -5. The time now is 20:37.


Powered by vBulletin®
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.
Copyright ©2004 - 2021, AppleNova