S0ImpulseCounter/S0Meters_py

S0Meters

Python program for logging data captured by DualImpCount (Arduino based S0 impulse counter module).

Intended to run (but not limited to) on a Raspberry Pi.

I developed this plus the hardware counter module because directly logging impulse counters with the Raspberry Pi was not reliable enough for me. Missed impulses, because the Pi was not running for some time, ghost impulses because the RasPi GPIOs seems somewhat unreliable (at least when reading them using Python), even when always using optocouplers on the impulse inputs.

Functions

• Data logging, implemented in 2 manners:
  • InfluxDB (current reading, current momentary power/usage)
  • file log (only daily minimum and total value)
• calculation of momentary power (or usage, depending on meter type) from measured time between impulses received from capturing module
• MQTT publishing of all current values
• calculation of todays/yesterdays total usage and publishing it over MQTT (only if file logging is enabled)
• configurable multiple InfluxDB instances (i.E. different DBs for a power meter and a gas meter and also for momentary (power) and long term (energy readings) data of the same meter)
• additional interval for energy readings, not to flood InfluxDB with data every few seconds as the hardware outputs data on every detected impulse (not implemented for momentary readings as there a high resolution is desired in the logs)

Installation

This example describes my setup - running on a Raspberry Pi 3/4 as user "pi".

1. extract s0meters folder to /home/pi/s0meters

2. install prerequisites:

   • Python 3

   • Python 3 modules:

     • pyserial
     • paho-mqtt
     • PyYAML

     • influxdb

       pip3 install pyserial paho-mqtt PyYAML influxdb
       

3. copy or symlink s0meters.service to /etc/systemd/system,

then reload systemctl and start service:

   sudo ln -s s0meters.service /etc/systemd/system
   or
   sudo cp s0meters.service /etc/systemd/system
   
   sudo systemctl daemon-reload
   sudo systemctl enable s0meters.service

1. edit configuration to your needs (see below)

2. when configuration is finished (and tested in interactive mode), start service:

   sudo systemctl start s0meters.service
   

3. check if it is still running after some time:

   sudo systemctl status s0meters.service
   

Configuration

Configuration is split in 3 parts. Base configuration is in ini format - most parameters are self describing

Main configuration

config file: s0meters.ini

[main]
# add timestamps to console output in debug/verbose mode
consoleAddTimestamp = true

# additional YAML config files
meters_config_yml = s0meters.yml
influx_config_yml = s0meters_influxdb.yml

[hardware]
serialPort = /dev/ttyUSB0
serialBaud = 57600
serialTout = 1

[filelog]
# needed for calculating today/yesterday totals, so only disable if that is not needed
enable = True

# storage path must be existing and writable for the user that runs s0meters.py (sub directories are created as needed)
storage_path = /home/pi/s0meters

[mqtt]
enable = true
server = mqtt.lan
port = 1883
keepalive = 60
user = mqttuser
password = *******

# MQTT topics
# Note: this is only for status infos and commands, meters Out-Topics must be defined in the meters configuration!
# In-Topic for sending commands to the device
topic_cmd = Top5/ImpCount/cmd
# Out-Topic
topic_stat = Top5/ImpCount/stat
# Out-Topic for response to commands (can be the same as topic_stat)
topic_cmdresponse = Top5/ImpCount/cmdResponse

Meters configuration

config file: meters.yml (unless differently defined in s0meters.ini)

In this YAML styled config file, all meters/counters are declared. The main identifier is the counter number used in the hardware counter module, so for "C1" the identifier is 1 and so on.

1: # meter ID as used in counter hardware (="C1")
  name: "Power"
  impPerUnit: 1000
  unit: "kWh"
  digits: 3
  readingType: "Energy_Total__kWh"
  momType: "power"
  momUnit: "W"
  momDigits: 0
  momFactor: 1000
  #conv_unit: "kWh"
  #conv_factor: 10.73184
  #conv_digits: 2
  cost_unit: "EUR"
  cost_per_unit: 0.330995
  cost_from_conv: False
  statTopic: "Powermeter"
  influxInstance_energy: "energymeters"
  influxInstance_mom: "energy_momentary"
  influxMeasurement_energy: "energy"
  influxMeasurement_mom: "energy"
  influxFieldName_energy: "Energy_Total__kWh"
  influxFieldName_mom: "Power__W"
  influxMinWriteInterval_energy: 300
  MQTTPublishJSON: False
  historyPublishInterval: 300
  billingStartDate: '2022-03-03'
2: # meter ID as used in counter hardware (="C2")
  name: "Gas"
  impPerUnit: 100
  unit: "m3"
  digits: 2
  #readingType: ""
  type: "usage"
  conv_unit: "kWh"
  conv_factor: 10.73184
  conv_digits: 2
  cost_unit: "EUR"
  cost_per_unit: 0.33
  cost_from_conv: True
  momType: "momUsage"
  momUnit: "m3/h"
  momDigits: 2
  momFactor: 1
  momType_conv1: "momUsage__kW"
  momUnit_conv1: "kW"
  momDigits_conv1: 2
  momFactor_conv1: 10.73184
  momType_conv2: "momUsage__l_min"
  momUnit_conv2: "l/min"
  momDigits_conv2: 1
  momFactor_conv2: 16.6667
  statTopic: "Gasmeter"
  influxInstance_energy: "energymeters"
  influxInstance_mom: "energy_momentary"
  influxMeasurement_energy: "gas"
  influxMeasurement_mom: "gas"
  influxFieldName_energy: "Gas_Total__m3"
  ##influxFieldName_energy_conv: "Gas_Total__kWh"
  influxFieldName_mom: "Gas_Usage__m3_h"
  #influxFieldName_mom_conv1: "Gas_Usage__kW"
  #influxFieldName_mom_conv2: "Gas_Usage__l_min"
  influxMinWriteInterval_energy: 300
  MQTTPublishJSON: False
  historyPublishInterval: 300
  billingStartDate: '2022-03-02'

• name: free text, used for path of file log and internally
• impPerUnit: integer, normally 10, 100 or 1000, used for calculation of momentary units
• unit: free text, but only A-Z a-z _ and - should be used. Used for MQTT topics Yesterday__unit and Today__unit
• digits: integer, number of digits in output of counter reading
• readingType: text string - used as topic name in MQTT output if defined, otherwise subtopic name = "reading"
• type: "usage" - not really used for anything right now
• conv_unit: unit for converted value, used for Today and Yesterday readings (currently not for Total)
• conv_factor: factor for conversion to a different unit
• conv_digits: digits to round convertion to
• cost_unit: currency for cost calculation, used in MQTT topic
• cost_per_unit: price of one energy-unit
• cost_from_conv: True or False - if True, cost will be calculated from the converted value, if False from the origin
• momType: type of momentary reading. used as MQTT sub-topic
• momUnit: free text, currently only used in MQTT JSON output
• momDigits: integer, number of digits in output of momentary value

• momFactor: factor for calculation of momentary value Formula for calculation: momValue = (3600000 / dTime / impPerUnit) * momFactor where dTime is the measured time between impulses in milliseconds. Normally 1, except if the unit of momentary value should be different - for example:

  • power meter, 1000 imp per unit = 1 kWh: momFactor=1000 -> output in Watts momFactor=1 -> output in kW
  • gas meter, 100 imp per unit = 1m³: momFactor=1 --> output in m³/h momFactor=1000 --> output in dm³/h (l/h)

• momType_conv1: type of converted momentary reading. used as MQTT sub-topic

• momUnit_conv1: free text, currently only used in MQTT JSON output

• momDigits_conv1: integer, number of digits in output of converted momentary value

• momFactor_conv1: factor for calculation of converted momentary value Formula for calculation: momValue_conv1 = momValue * momFactor_conv1 where momValue is the origin unconverted value. For example:

  • gas meter, convert m³/h to kW: factor of ~10
  • gas meter, convert m³/h to l/min: factor of 16.6667

• momType_conv2: type of 2nd converted momentary reading. used as MQTT sub-topic

• momUnit_conv2: free text, currently only used in MQTT JSON output

• momDigits_conv2: integer, number of digits in output of 2nd converted momentary value

• momFactor_conv2: factor for calculation of 2nd converted momentary value Formula for calculation: momValue_conv2 = momValue * momFactor_conv2 where momValue is the origin unconverted value. For example:

  • gas meter, convert m³/h to kW: factor of ~10
  • gas meter, convert m³/h to l/min: factor of 16.6667

• statTopic: MQTT topic prefix to publish all values of this meter i.E. if statTopic = "Powermeter", following topics will be published to:

  • Powermeter/reading = current counter reading (total)
  • Powermeter/today = increase of counter reading today
  • Powermeter/yesterday = increase of counter reading yesterday
  • Powermeter/[momType] = current value of momentary value ("power" for example 1, "usage" for example 2)
  • Powermeter/json = JSON formatted output containing all values

• influxInstance_energy: a valid InfluxDB instance from s0meters_influxdb.yml where energy/counter readings are written to

• influxInstance_mom: a valid InfluxDB instance from s0meters_influxdb.yml where momentary values are written to

• influxMeasurement_energy: InfluxDB measurement name for energy (total counter value)

• influxMeasurement_mom: InfluxDB measurement name for momentary values

• influxFieldName_energy: InfluxDB field name for energy (total counter value)

• influxFieldName_energy_conv: InfluxDB field name for converted energy (total counter value) - CURRENTLY NOT IMPLEMENTED

• influxFieldName_mom: InfluxDB measurement name for momentary value

• influxFieldName_mom_conv1: InfluxDB measurement name for converted momentary value

• influxFieldName_mom_conv2: InfluxDB measurement name for 2nd converted momentary value

• influxMinWriteInterval_energy: minimal InfluxDB write interval for energy readings If no interval is configured, EVERY impulse received will be written to InfluxDB! There is no such option for the momentary value, as there a high resolution is desired in the logs.

• MQTTPublishJSON: if set to True, a JSON object with all data is published on MQTT every time

• historyPublishInterval: interval in seconds to publish history data (yesterday, this week, this month etc.)

• billingStartDate: date on which the current billing period starts. Format: YYYY-MM-DD. Used for statistics (consumption since last bill)

InfluxDB configuration

config file: s0meters_influx.yml (unless differently defined in s0meters.ini)

• It is possible to define as many InfluxDB instances as needed
• Every instance has its identifier which must be unique
• It is only necessary to define different instances if the database name or server is different. If only the measurement name differs, that can be set in the meters configuration for each meter
• defaultMeasurement is not really necessary as normally the used measurement name should be defined for each meter in the meters configuration YAML

• All other parameters here are self describing. Comment out/remove username/password if the server does not use it.

  energymeters:
  host: localhost
  port: 8086
  username: s0meters
  password: ********
  database: energymeters
  defaultMeasurement: energy
  energy_momentary:
  host: localhost
  port: 8086
  username: s0meters
  password: ********
  database: energy_momentary
  defaultMeasurement: energy
  
  

controlarduino.py, flasharduino.sh

Tools for serial monitor and to reset and flash Arduino board that supports it with reset [ms]command.

controlarduino.py is used by flasharduino.sh to reset the MCU.

This alternate reset/flash method can be used for Arduino based projects with:

• no direct USB connection
• UART connection available, but without DTS line (used by Arduino for resetting into bootloader), regardless of the reason
• when UART DTS is intentionally disconnected to prevent from sudden reset, which could lead to data loss (such as in this project with the counters eventually not yet stored to EEPROM)

In this project a reset [ms] command is built into the Arduino software which resets the hardware by pulling /RESET to low using a dedicated (normally INPUT) GPIO pin. This resets the Arduino in such a way, that it will start up in bootloader mode and can be flashed using avrdude. This reset command also calls a save-all-data-to-EEPROM routine before doing the actual reset.

flasharduino.sh sends the reset command using controlarduino.py, flashes the MCU using avrdude and then starts controlarduino.py in --monitor mode.