S0Meters

Python program for logging data captured by DualImpCount (Arduino based S0 impulse counter module).

Intended to run (but not limited to) on a Raspberry Pi.

I developed this plus the hardware counter module because directly logging impulse counters with the Raspberry Pi was not reliable enough for me. Missed impulses, because the Pi was not running for some time, ghost impulses because the RasPi GPIOs seems somewhat unreliable (at least when reading them using Python), even when always using optocouplers on the impulse inputs.

Functions

Data logging, implemented in 2 manners:
- InfluxDB (current reading, current momentary power/usage)
- file log (only daily minimum and total value)
calculation of momentary power (or usage, depending on meter type) from measured time between impulses received from capturing module
MQTT publishing of all current values
calculation of todays/yesterdays total usage and publishing it over MQTT (only if file logging is enabled)
configurable multiple InfluxDB instances (i.E. different DBs for a power meter and a gas meter and also for momentary (power) and long term (energy readings) data of the same meter)
additional interval for energy readings, not to flood InfluxDB with data every few seconds as the hardware outputs data on every detected impulse (not implemented for momentary readings as there a high resolution is desired in the logs)

Installation

This example describes my setup - running on a Raspberry Pi 3/4 as user "pi".

extract s0meters folder to /home/pi/s0meters
install prerequisites:
- Python 3
- Python 3 modules:
  - pyserial
  - paho-mqtt
  - PyYAML
  - influxdb
```
pip3 install pyserial paho-mqtt PyYAML influxdb
```
copy or symlink s0meters.service to /etc/systemd/system,

then reload systemctl and start service:

   sudo ln -s s0meters.service /etc/systemd/system
   or
   sudo cp s0meters.service /etc/systemd/system
   
   sudo systemctl daemon-reload
   sudo systemctl enable s0meters.service

edit configuration to your needs (see below)
when configuration is finished (and tested in interactive mode), start service:
```
sudo systemctl start s0meters.service
```
check if it is still running after some time:
```
sudo systemctl status s0meters.service
```

Configuration

Configuration is split in 3 parts. Base configuration is in ini format - most parameters are self describing

Main configuration

config file: s0meters.ini

[main]
# add timestamps to console output in debug/verbose mode
consoleAddTimestamp = true

# additional YAML config files
meters_config_yml = s0meters.yml
influx_config_yml = s0meters_influxdb.yml

[hardware]
serialPort = /dev/ttyUSB0
serialBaud = 57600
serialTout = 1

[filelog]
# needed for calculating today/yesterday totals, so only disable if that is not needed
enable = True

# storage path must be existing and writable for the user that runs s0meters.py (sub directories are created as needed)
storage_path = /home/pi/s0meters

[mqtt]
enable = true
server = mqtt.lan
port = 1883
keepalive = 60
user = mqttuser
password = *******

# MQTT topics
# Note: this is only for status infos and commands, meters Out-Topics must be defined in the meters configuration!
# In-Topic for sending commands to the device
topic_cmd = Top5/ImpCount/cmd
# Out-Topic
topic_stat = Top5/ImpCount/stat
# Out-Topic for response to commands (can be the same as topic_stat)
topic_cmdresponse = Top5/ImpCount/cmdResponse

Meters configuration

config file: meters.yml (unless differently defined in s0meters.ini)

In this YAML styled config file, all meters/counters are declared. The main identifier is the counter number used in the hardware counter module, so for "C1" the identifier is 1 and so on.

1: # meter ID as used in counter hardware (="C1")
  name: "Power"
  impPerUnit: 1000
  unit: "kWh"
  digits: 3
  readingType: "Energy_Total__kWh"
  momType: "power"
  momUnit: "W"
  momDigits: 0
  momFactor: 1000
  #conv_unit: "kWh"
  #conv_factor: 10.73184
  #conv_digits: 2
  cost_unit: "EUR"
  cost_per_unit: 0.330995
  cost_from_conv: False
  statTopic: "Powermeter"
  influxInstance_energy: "energymeters"
  influxInstance_mom: "energy_momentary"
  influxMeasurement_energy: "energy"
  influxMeasurement_mom: "energy"
  influxFieldName_energy: "Energy_Total__kWh"
  influxFieldName_mom: "Power__W"
  influxMinWriteInterval_energy: 300
  MQTTPublishJSON: False
  historyPublishInterval: 300
  billingStartDate: '2022-03-03'
2: # meter ID as used in counter hardware (="C2")
  name: "Gas"
  impPerUnit: 100
  unit: "m3"
  digits: 2
  #readingType: ""
  type: "usage"
  conv_unit: "kWh"
  conv_factor: 10.73184
  conv_digits: 2
  cost_unit: "EUR"
  cost_per_unit: 0.33
  cost_from_conv: True
  momType: "momUsage"
  momUnit: "m3/h"
  momDigits: 2
  momFactor: 1
  momType_conv1: "momUsage__kW"
  momUnit_conv1: "kW"
  momDigits_conv1: 2
  momFactor_conv1: 10.73184
  momType_conv2: "momUsage__l_min"
  momUnit_conv2: "l/min"
  momDigits_conv2: 1
  momFactor_conv2: 16.6667
  statTopic: "Gasmeter"
  influxInstance_energy: "energymeters"
  influxInstance_mom: "energy_momentary"
  influxMeasurement_energy: "gas"
  influxMeasurement_mom: "gas"
  influxFieldName_energy: "Gas_Total__m3"
  ##influxFieldName_energy_conv: "Gas_Total__kWh"
  influxFieldName_mom: "Gas_Usage__m3_h"
  #influxFieldName_mom_conv1: "Gas_Usage__kW"
  #influxFieldName_mom_conv2: "Gas_Usage__l_min"
  influxMinWriteInterval_energy: 300
  MQTTPublishJSON: False
  historyPublishInterval: 300
  billingStartDate: '2022-03-02'

name: free text, used for path of file log and internally
impPerUnit: integer, normally 10, 100 or 1000, used for calculation of momentary units
unit: free text, but only A-Z a-z _ and - should be used. Used for MQTT topics Yesterday__unit and Today__unit
digits: integer, number of digits in output of counter reading
readingType: text string - used as topic name in MQTT output if defined, otherwise subtopic name = "reading"
type: "usage" - not really used for anything right now
conv_unit: unit for converted value, used for Today and Yesterday readings (currently not for Total)
conv_factor: factor for conversion to a different unit
conv_digits: digits to round convertion to
cost_unit: currency for cost calculation, used in MQTT topic
cost_per_unit: price of one energy-unit
cost_from_conv: True or False - if True, cost will be calculated from the converted value, if False from the origin
momType: type of momentary reading. used as MQTT sub-topic
momUnit: free text, currently only used in MQTT JSON output
momDigits: integer, number of digits in output of momentary value
momFactor: factor for calculation of momentary value Formula for calculation: momValue = (3600000 / dTime / impPerUnit) * momFactor where dTime is the measured time between impulses in milliseconds. Normally 1, except if the unit of momentary value should be different - for example:
- power meter, 1000 imp per unit = 1 kWh: momFactor=1000 -> output in Watts momFactor=1 -> output in kW
- gas meter, 100 imp per unit = 1m³: momFactor=1 --> output in m³/h momFactor=1000 --> output in dm³/h (l/h)
momType_conv1: type of converted momentary reading. used as MQTT sub-topic
momUnit_conv1: free text, currently only used in MQTT JSON output
momDigits_conv1: integer, number of digits in output of converted momentary value
momFactor_conv1: factor for calculation of converted momentary value Formula for calculation: momValue_conv1 = momValue * momFactor_conv1 where momValue is the origin unconverted value. For example:
- gas meter, convert m³/h to kW: factor of ~10
- gas meter, convert m³/h to l/min: factor of 16.6667
momType_conv2: type of 2nd converted momentary reading. used as MQTT sub-topic
momUnit_conv2: free text, currently only used in MQTT JSON output
momDigits_conv2: integer, number of digits in output of 2nd converted momentary value
momFactor_conv2: factor for calculation of 2nd converted momentary value Formula for calculation: momValue_conv2 = momValue * momFactor_conv2 where momValue is the origin unconverted value. For example:
- gas meter, convert m³/h to kW: factor of ~10
- gas meter, convert m³/h to l/min: factor of 16.6667
statTopic: MQTT topic prefix to publish all values of this meter i.E. if statTopic = "Powermeter", following topics will be published to:
- Powermeter/reading = current counter reading (total)
- Powermeter/today = increase of counter reading today
- Powermeter/yesterday = increase of counter reading yesterday
- Powermeter/[momType] = current value of momentary value ("power" for example 1, "usage" for example 2)
- Powermeter/json = JSON formatted output containing all values
influxInstance_energy: a valid InfluxDB instance from s0meters_influxdb.yml where energy/counter readings are written to
influxInstance_mom: a valid InfluxDB instance from s0meters_influxdb.yml where momentary values are written to
influxMeasurement_energy: InfluxDB measurement name for energy (total counter value)
influxMeasurement_mom: InfluxDB measurement name for momentary values
influxFieldName_energy: InfluxDB field name for energy (total counter value)
influxFieldName_energy_conv: InfluxDB field name for converted energy (total counter value) - CURRENTLY NOT IMPLEMENTED
influxFieldName_mom: InfluxDB measurement name for momentary value
influxFieldName_mom_conv1: InfluxDB measurement name for converted momentary value
influxFieldName_mom_conv2: InfluxDB measurement name for 2nd converted momentary value
influxMinWriteInterval_energy: minimal InfluxDB write interval for energy readings If no interval is configured, EVERY impulse received will be written to InfluxDB! There is no such option for the momentary value, as there a high resolution is desired in the logs.
MQTTPublishJSON: if set to True, a JSON object with all data is published on MQTT every time
historyPublishInterval: interval in seconds to publish history data (yesterday, this week, this month etc.)
billingStartDate: date on which the current billing period starts. Format: YYYY-MM-DD. Used for statistics (consumption since last bill)

InfluxDB configuration

config file: s0meters_influx.yml (unless differently defined in s0meters.ini)

It is possible to define as many InfluxDB instances as needed
Every instance has its identifier which must be unique
It is only necessary to define different instances if the database name or server is different. If only the measurement name differs, that can be set in the meters configuration for each meter
defaultMeasurement is not really necessary as normally the used measurement name should be defined for each meter in the meters configuration YAML

All other parameters here are self describing. Comment out/remove username/password if the server does not use it.

energymeters:
host: localhost
port: 8086
username: s0meters
password: ********
database: energymeters
defaultMeasurement: energy
energy_momentary:
host: localhost
port: 8086
username: s0meters
password: ********
database: energy_momentary
defaultMeasurement: energy

controlarduino.py, flasharduino.sh

Tools for serial monitor and to reset and flash Arduino board that supports it with reset [ms]command.

controlarduino.py is used by flasharduino.sh to reset the MCU.

This alternate reset/flash method can be used for Arduino based projects with:

no direct USB connection
UART connection available, but without DTS line (used by Arduino for resetting into bootloader), regardless of the reason
when UART DTS is intentionally disconnected to prevent from sudden reset, which could lead to data loss (such as in this project with the counters eventually not yet stored to EEPROM)

In this project a reset [ms] command is built into the Arduino software which resets the hardware by pulling /RESET to low using a dedicated (normally INPUT) GPIO pin. This resets the Arduino in such a way, that it will start up in bootloader mode and can be flashed using avrdude. This reset command also calls a save-all-data-to-EEPROM routine before doing the actual reset.

flasharduino.sh sends the reset command using controlarduino.py, flashes the MCU using avrdude and then starts controlarduino.py in --monitor mode.

README.md 13 KB History Raw