User Name
Password
AppleNova Forums » Programmer's Nook »

Custom MQTT publishing for weather collection


Register Members List Calendar Search FAQ Posting Guidelines
Custom MQTT publishing for weather collection
Thread Tools
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2023-11-06, 16:46

Don't worry, I know few people on this forum will care about this but I REALLY want to share it because I have figured out how to make something happen I've been want to for a long time!

I have 6 Raspberry Pis around my place that have a temp and humidity sensor connected to the GPIO pins. On each there is a python script that runs reading the data and writing the results to two text files in /tmp for other services to collect this data. I did it this way because the data isn't consistent and occasionally it will fail to update resulting in the collection points to barf all over the data. By writing to the text files I'm able to prevent this occasional failure. So the results look like this:

Code:
$ cat /tmp/temp 75.7 $ cat /tmp/humidity 54.6
Currently this data is all feeding my local Zabbix install and data logging there. I'm going to move away from this to Home Assistant for the data logging since I'm already using it for certain functions in my home. This means I need to get this data into an MQTT/JSON format. This is how I'm reading cheap rtl_433 that put out data like this:
Code:
{ "time": "2023-11-06 21:16:36", "model": "Oregon-THGR810", "id": 115, "channel": 1, "battery_ok": 0, "temperature_C": 22, "humidity": 50 }
So in order to get this data from the Pi into the HA install I've written this script that take variables and then publishes the data to the MQTT broker in my HA install. Here is the script:
Code:
#!/bin/bash datestring=$(date '+%Y-%m-%d %k:%M:%S') temp=$(/usr/bin/cat /tmp/temp) humidity=$(/usr/bin/cat /tmp/humidity) mqttid=$(/usr/bin/cat /proc/cpuinfo|grep Serial|cut -d: -f2|rev|cut -c 1-4|rev) mqttuser=$(/usr/bin/cat /home/turtle/.mqttuser) mqttpasswd=$(/usr/bin/cat /home/turtle/.mqttpasswd) mqttmessage=$"{ \"time\": \"$datestring\", \"model\": \"rpiweather\", \"id\": \"$mqttid\", \"temperature_F\": $temp, \"humidity\": $humidity }" #echo $mqttmessage mosquitto_pub -h 10.10.0.12 -u $mqttuser -P $mqttpasswd -t "rpidata/rpiweather/$mqttid" -m "$mqttmessage"
This is what the output looks like:
Code:
{ "time": "2023-11-06 16:18:37", "model": "rpiweather", "id": "47ce", "temperature_F": 75.7, "humidity": 54.8 }
Each data source needs to be uniquely identified and to do this I'm using the last four character of the Pi Serial number, that matches the same for the MAC address.

Since my MQTT instance is password protected, I've placed the password files outside of the script so it can be updated as needed and not require the password to be entered in plain text. I'll lock down the permissions once I push this script to full production based on the user account I have running the cron to push this data. Likely it will just push once per minute since weather data shouldn't change too much minute by minute.

This is all really in preps for my newest piece of hardware: a BMP180. This will allow me to start data logging the barometric pressure. I'm going to do a similar function of writing the data to disk and then pull from that file. I don't know how stable the data output is from the BMP180, but I'm expecting it will be like the DHT22 for the temp/humidity.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
drewprops
Space Pirate
 
Join Date: May 2004
Location: Atlanta
 
2023-11-06, 20:49

Will any of this feed a screen so you have up to date in a heads up fashion?

How do you get away with a space after "/cat"?


...
  quote
psmith2.0
Mr. Vieira
 
Join Date: May 2004
Location: Tennessee
 
2023-11-07, 00:31

I’m not a dumb guy. I’ve got a fairly good sense of most things. I can wrap my head around nearly anything and feel reasonably sure that I can understand, explain or discuss things in my world.

It’s not often I run across a thread here where I understand/grasp absolutely nothing about what’s being said.

Congrats, Tony!

I think it’s awesome people know these kinds of things and are doing so much more with their knowledge and hardware that I ever do. I really do.

Gonna buy me an Etch-a-Sketch after tacos later today/tomorrow!

Last edited by psmith2.0 : 2023-11-07 at 06:32.
  quote
chucker
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2023-11-07, 05:55

TL;DR, I think: Turtle has a bunch of sensors that collect hyperlocal weather data, and made a custom setup to aggregate them, probably ultimately in HomeKit.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2023-11-07, 13:55

Quote:
Originally Posted by drewprops View Post
Will any of this feed a screen so you have up to date in a heads up fashion?

How do you get away with a space after "/cat"?


...
Did you notice the "/cat" was right before "/home", that is because cats need their space to feel like they have freedom. If you pick it up and put it in a box it will jump out, but if you leave an open box (the smaller the better) they will jump right in! So their space facilitates this in the command.

Quote:
Originally Posted by psmith2.0 View Post
I’m not a dumb guy. I’ve got a fairly good sense of most things. I can wrap my head around nearly anything and feel reasonably sure that I can understand, explain or discuss things in my world.

It’s not often I run across a thread here where I understand/grasp absolutely nothing about what’s being said.

Congrats, Tony!

I think it’s awesome people know these kinds of things and are doing so much more with their knowledge and hardware that I ever do. I really do.

Gonna buy me an Etch-a-Sketch after tacos later today/tomorrow!
It is always nice when someone can help get a new hobby to learn! See, some day with enough work you will be able to understand what I wrote!

Of course, I'm guessing this is just not going to happen, so you can live vicariously through my coding projects.

Quote:
Originally Posted by chucker View Post
TL;DR, I think: Turtle has a bunch of sensors that collect hyperlocal weather data, and made a custom setup to aggregate them, probably ultimately in HomeKit.
Yeah something like that. I'm going to eventually branch beyond weather data, but getting it from one format to another with languages I know was something I wanted to accomplish. Using bash also limits dependency/compatibility and upgrade/update problem. I do want the data in HomeKit for sure. Right now it will all just feed Home Assistant. I do plan to have Home Assistant feed an InfluxDB and then use Grafana to display the data again. One step at a time. Getting this original data in to MQTT is the first step though.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
psmith2.0
Mr. Vieira
 
Join Date: May 2004
Location: Tennessee
 
2023-11-07, 15:45

Quote:
Originally Posted by turtle View Post
It is always nice when someone can help get a new hobby to learn! See, some day with enough work you will be able to understand what I wrote!

Of course, I'm guessing this is just not going to happen, so you can live vicariously through my coding projects.
Yeah, that's about my speed/capability, at this point. It's just neat to observe things I know nothing about, and then head off to Google, Wikipedia and YouTube to fill in the blanks.
  quote
kscherer
Which way is up?
 
Join Date: Aug 2004
Location: Boyzeee
 
2023-11-07, 16:25

Tony and Brad are forever posting gibberish.

I think it's a secret code as they work out their evil-genius world-domination plan.
  quote
Ebby
Subdued and Medicated
 
Join Date: May 2004
Location: Over Yander
Send a message via AIM to Ebby  
2023-11-07, 17:11

I'm not too far behind. Set up Home Assistant OS (outgrew the docker image) and my SO and I programmed a ESP8266 and ESP32-C3 to control Halloween lighting and church decorations using ESPhome and WLED.

We have quite a lot of smart stuff in our home that was, surprisingly, detected immediately. Lights, switches, timers, entertainment, security, servers... heck, even my utilities (after granting access)! A few things needed config and a couple still needed manual installs. But WOW!

I just started learning about MQTT stuff, but haven't made anything work yet. I know I've only scratched the surface because nothing seems too complicated or snowballing into a monumental undertaking yet.

^^ One more quality post from the desk of Ebby. ^^
SSBA | SmockBogger | SporkNET
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2023-11-08, 10:22

In my first install of HA I did it using a container because I was "just playing" with it. Then I found the Span Panel and Tesla Gateway add-ons and realized I had to go all in really. Over in the Residential Solar thread I talked about automating my home going off-grid during peak. This is actually still going on an I'm loving the automation. No more accidental peak rate consumption... at all.

So I went and bought a rPi4 (4GB) just to run my HAOS install with a PoE HAT and SSD for boot drive:


Now I have room to move and plenty of CPU to handle anything I throw at it. This morning I added the script to my gatekeeper reverse proxy with the DHT22 on it and this is what my sensor section of the configuration.yml looks like:
Code:
mqtt: sensor: - name: Network Rack Temperature unique_id: 6b9c_rpiweather_temperature device_class: temperature unit_of_measurement: '°F' value_template: '{{ value_json.temperature_F }}' state_topic: rpidata/rpiweather/6b9c json_attributes_topic: rpidata/rpiweather/6b9c - name: Network Rack Humidity unique_id: 6b9c_rpiweather_humidity device_class: humidity unit_of_measurement: '%' value_template: '{{ value_json.humidity }}' state_topic: rpidata/rpiweather/6b9c json_attributes_topic: rpidata/rpiweather/6b9c
Adding those as sensors in a "Climate" Dashboard I have data logged looking like this:



Directly related to the creation of this thread, I'm able to take my script and copy it to the rPis and create the .mqttuser and .mqttpasswd files and set up the cron jobs. Of course, I'm doing it in a wacky way, but I'm sending data every 30 seconds:
Code:
* * * * * /home/pi/bin/climatemqtt.sh * * * * * ( sleep 30 ; /home/pi/bin/climatemqtt.sh )
So professional. I will say though, I'm doing this simplistically for a reason. I'm doing this in my off time for fun and I don't want to have to dig to figure out how I did it again later once something changes and I have to figure it out.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2023-11-08, 10:52

Next up is to solder these pin on this sensor and get it working on my "pisniffer" that is running a SDR pulling my 433Mhz sensors into this MQTT world for HA to use!


Once I get the data from it I'll be able to feed barometric pressure into the HA for graphing and data collection.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
Ebby
Subdued and Medicated
 
Join Date: May 2004
Location: Over Yander
Send a message via AIM to Ebby  
2023-11-11, 04:17

*Record scratch*

Whoa, a... PiRack?

*Chefs Kiss* What a beautiful thing I didn't know I needed!
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2023-11-13, 11:00

Quote:
Originally Posted by Ebby View Post
*Record scratch*

Whoa, a... PiRack?

*Chefs Kiss* What a beautiful thing I didn't know I needed!
Yeah, I put of getting one for a while and finally just did it. I'm so glad too. Paired with a SATA/USB SSD and PoE HAT it is awesome. Even the one I'm running my rtl_433 sniffer on that isn't PoE (rPi 3) is still better in this rack. Organizing those permanent members of my network is just simple now.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2023-11-13, 11:16

In my next phase I have soldered the terminal posts on the sensor and connected it to "pisniffer" that is running the rtl_433 software grabbing my weather sensor data from my various sensors around the house, including an Acurite Atlas on my roof.

I have the sensor wires running to the front so I can see it, though it could be hidden behind the front. I like knowing it is there, plus it has better ventilation in the front and will be a closer to accurate temp reading. The temp reading is only for the pressure calculation though. This is the example script provided with the code though I have modified the bottom section to allow me to do some math and get the pressure in "inHg" instead. I'm still going to have to get this data to MQTT for Home Assistant though. That is next.

Code:
turtle@pisniffer:~ $ cat bin/pressure.py #!/usr/bin/python import Adafruit_BMP.BMP085 as BMP085 sensor = BMP085.BMP085() # You can also optionally change the BMP085 mode to one of BMP085_ULTRALOWPOWER, # BMP085_STANDARD, BMP085_HIGHRES, or BMP085_ULTRAHIGHRES. See the BMP085 # datasheet for more details on the meanings of each mode (accuracy and power # consumption are primarily the differences). The default mode is STANDARD. sensor = BMP085.BMP085(mode=BMP085.BMP085_ULTRAHIGHRES) print('Temp = {0:0.2f} *C'.format(sensor.read_temperature())) print('Pressure = {0:0.2f} Pa'.format(sensor.read_pressure())) print('Altitude = {0:0.2f} m'.format(sensor.read_altitude())) print('Sealevel Pressure = {0:0.2f} Pa'.format(sensor.read_sealevel_pressure())) absolutePressure = int(format(sensor.read_pressure())) relativePressure = ( absolutePressure / 0.969165 ) print(relativePressure) pressure = relativePressure * .00029527578 print(pressure) turtle@pisniffer:~ $ python bin/pressure.py Temp = 28.90 *C Pressure = 100413.00 Pa Altitude = 75.87 m Sealevel Pressure = 100424.00 Pa 103617.03115568556 30.595599695779352
I still have to get the math that adjusts the pressure based on the temperature... still digging for that one. The 30.59 is close to what my Acurite says it is, but there is about a .14inHg difference between the two right now. That isn't much, but I'm not sure which one is most accurate right now. I'm assuming the one in the Acurite, but it might actually be my sensor on the rPi.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2023-11-13, 20:14

My calculations are off for the barometric pressure. It seems to be a solid .15inHg off from the one in my weather station I also compared it to the ones in the local area that feed wunderground. While there is a little difference from my sensor, those in wunderground are close to my weather station. So I have to figure out why my new BMP180 is off. I’m sure there is a calculation I need to do with my absolute measurement I’m getting, I just don’t know what it is.

If it is consistently off by .15inHg then I might just add that to my calculations instead and go from there. It will be close enough.

Really though, I want it right.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2023-11-21, 19:04

I have the basic python script running to get the absolute pressure now:
Code:
turtle@pisniffer:~/bin $ cat writePressure.py #!//usr/bin/python3 # SPDX-FileCopyrightText: 2021 ladyada for Adafruit Industries # SPDX-License-Identifier: MIT import time import board import Adafruit_BMP.BMP085 as BMP085 # You can also optionally change the BMP085 mode to one of BMP085_ULTRALOWPOWER, # BMP085_STANDARD, BMP085_HIGHRES, or BMP085_ULTRAHIGHRES. See the BMP085 # datasheet for more details on the meanings of each mode (accuracy and power # consumption are primarily the differences). The default mode is STANDARD. sensor = BMP085.BMP085(mode=BMP085.BMP085_ULTRAHIGHRES) while True: try: # Print the values to the serial port pressure = sensor.read_pressure() print('Pressure = {0:0.2f} Pa'.format(sensor.read_pressure())) with open('/tmp/pressure', 'w') as f: f.write("{0:0.2f}".format( pressure ) ) except RuntimeError as error: # Errors happen fairly often, DHT's are hard to read, just keep going print(error.args[0]) time.sleep(2.0) continue except Exception as error: sensor.exit() raise error time.sleep(5.0) turtle@pisniffer:~/bin $ cat /tmp/pressure 98654.00
Now what I need to do is figure out how to get the python script to just send the MQTT message directly from the script rather than make a second script to read the text file value and then send that. For now that is what I'm going to have to do... but since the script is already grabbing the data it just makes sense for it to send the data too.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2023-11-21, 20:49

For the moment, I did set up the script I started this thread off with to send my data to HA.
Code:
turtle@pisniffer:~/bin $ /home/turtle/bin/climatemqtt.sh { "time": "2023-11-21 19:22:02", "model": "rpiweather", "id": "47ce", "pressure": 98700.00 }
It works perfectly, though it makes sense to just bake it into the python script itself.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
drewprops
Space Pirate
 
Join Date: May 2004
Location: Atlanta
 
2023-11-22, 20:40

Did you solve the barometric issue or just do a hard code adjustment?

Do you have an alert to your phone?

What about water sensors? Like, if a water heater tank failed?

...
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2023-11-22, 22:50

I did not get the pressure adjusted properly for sea level. At this point I'm just going to leave it in "absolute" value for my current pressure. At some point I will get it corrected for sea level so it matches the rest of the readings in my region. When I do, it will be a hard coded formula. That is all it is anyway, I just haven't dug into that formula or how to properly implement it. I'm also logging as Pa instead of inHg that we use here in the states:




So I can get alerts of my phone if I set up trigger points and such, but I won't.

As for leak detection and such, I can do that with HomeKit but haven't. I should, but haven't.

Louis L'Amour, “To make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
Posting Rules Navigation
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Post Reply

Forum Jump
Thread Tools
Similar Threads
Thread Thread Starter Forum Replies Last Post
Atari's Greatest Hits collection Xaqtly Third-Party Products 5 2011-04-07 18:09
World's Greatest Music Collection Brave Ulysses AppleOutsider 9 2008-02-22 06:48
mp3 collection merging ast3r3x Genius Bar 3 2007-11-16 01:38
iTunes 5 manages your PDF collection! gedto Apple Products 3 2005-09-10 14:32
Very Nice Collection garum General Discussion 2 2005-07-15 06:47


« Previous Thread | Next Thread »

All times are GMT -5. The time now is 07:35.


Powered by vBulletin®
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004 - 2024, AppleNova