initial commit of extended version, see CHANGELOG.md for details

CHANGELOG.md

@@ -1,4 +1,35 @@
 ## Change Log
 
+
+## v0.2 - 2020-10-01
+* rename to ModbusLogMQTT
+* Python 3 compatibility
+* added app-configuration using ini file - instruments configuration keeps in yml files
+* added MQTT client, publishing all read data on configured topics
+* added simple file logging
+  * writes 2 files per meter and day on each date rollover: 
+    * today min - kWh total from meter at that time
+    * yesterday total - calculated from current kWh reading and yesterdays saved value
+    * both values are also published on every energy reading interval to MQTT and written into InfluxDB if desired
+    * this is meant either as a backup of the most important data (I lost many data in the past due to InfluxDB misconfiguration and bad backup), and in order to calculate and publish daily energy usage directly and without complicated database transactions (I used to display "energy today" and "energy yesterday" on a simple display that just subscribes a MQTT topic and displays what it gets)
+* changed repeat method to run first iteration immediately after start
+* stability improvments
+  * do not exit script with error on repeated instrument read errors, instead send error msg on MQTT
+* heavy rework of meters configuration and reading method:
+  * split data aquiring in different intervals, where the base interval (which runs the main method) is now intended to be very short. All other intervals are based on this (just measuring elapsed time since last run within main method)
+  * data aquiring run as often as possible in order to get meaningful momentary (power, current...) data
+  * meter types configurations:
+    * split in 2 sections: momentary and energy
+    * each reading now has 2 parameters: address, decimals - so every reading can have it´s own reasonable conversion, i.e. 0 decimals for power reading, as its not that accurate anyway, or 3 decimals for a kWh reading. 0 decimals values are converted to int for MQTT to prevent trailing zeros, but not for InfluxDB as it breaks data writing when now-integer-values after config changes are already stored in InfluxDB as float
+  * meters and meter types configuration is no more checked for file update on every iteration, instead in a fixed interval of 60s, meter types configuration is read on demand (if a meter with this type is configured) and stored in memory rather than reading the yml file on every iteration
+  * momentary readings are aquired on every iteration of the main method with no or very short interval
+    * on significant change in power reading: immediately publish on MQTT / write to InfluxDB, otherwise in a fixed additional interval
+  * energy readings are only aquired and processed in an additional, longer interval, which also handles file logging and yesterday total calculation method
+* InfluxDB
+  * split to 2 different databases for momentary/energy data if desired, as it makes sense to store power readings and energy readings in different databases with different retention policies and continuous queries
+
+
+## forked - 2020-09-30
+
 ## [0.1] - 2017-11-09
 * Read registers of RS485 modbus based energy meter 

README.md

@@ -1,104 +1,39 @@
-# Energy Meter Logger
-Log your Energy Meter data on a Raspberry Pi and plot graphs of your energy consumption.
-Its been verified to work with a Raspberry Pi with a Linksprite RS485 shield and reading values from WEBIQ131D / SDM120 and WEBIQ343L / SDM630. By changing the meters.yml file and making a corresponding [model].yml file it should be possible to use other modbus enabled models.
+# Modbus Energy Meter Logger and MQTT gateway
+Log Modbus Energy Meter data to InfluxDB on a Raspberry Pi and publish values via MQTT 
+
+Based on original project on [Github](https://github.com/samuelphy/energy-meter-logger)
+
+#### Added features
+
+* base configuration using ini file
+* MQTT publishing to use the readings in other systems
+* split data aquiring completely in momentary (power) and energy readings with different intervals
+* calculate daily total energy usage and log to file system as a backup
+* higher sample rate for momentary power reading, write to database on power changes and/or interval
+* separate interval for aquiring/writing energy readings
+* split InfluxDB logging in momentary (power) and energy readings (seperate databases if desired to enable usage of different retention policies and continuous queries)
+* enhanced meters configuration to support that changes, using yaml file as in original project
+* many more improvements
+
+Verified to work on a Raspberry Pi 4 with Digitus USB-RS485 Interface, reading values from 3 Eastron SDM120 instruments. By changing the meters.yml file and making a corresponding metertype_[model].yml file it should be possible to use other modbus enabled models.
 
 ### Requirements
 
 #### Hardware
 
-* Raspberry Pi 3
-* [Linksprite RS485 Shield V3 for RPi](http://linksprite.com/wiki/index.php5?title=RS485/GPIO_Shield_for_Raspberry_Pi_V3.0)
-* Modbus based Energy Meter, e.g WEBIQ 131D / Eastron SDM120 or WEBIQ 343L / Eastron SMD630
+* Raspberry Pi 2/3/4
+* RS485 USB interface or RS485 Shield for RPi
+* Modbus based Energy Meter(s), e.g WEBIQ 131D / Eastron SDM120 or WEBIQ 343L / Eastron SMD630
 
 #### Software
 
 * Rasbian
-* Python 2.7 and PIP
+* Python 3.7 and PIP
 * [Minimalmodbus](https://minimalmodbus.readthedocs.io/en/master/)
-* [InfluxDB](https://docs.influxdata.com/influxdb/v1.3/)
+* [InfluxDB](https://docs.influxdata.com/influxdb/v1.8/)
 * [Grafana](http://docs.grafana.org/)
 
 ### Prerequisite
 
-This project has been documented at [Hackster](https://www.hackster.io/samuelphy/energy-meter-logger-6a3468). Please follow the instructions there for more detailed information.
-
-### Installation
-#### Install InfluxDB*
-
-##### Step-by-step instructions
-* Add the InfluxData repository
-    ```sh
-    $ curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
-    $ source /etc/os-release
-    $ test $VERSION_ID = "9" && echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
-    ```
-* Download and install
-    ```sh
-    $ sudo apt-get update && sudo apt-get install influxdb
-    ```
-* Start the influxdb service
-    ```sh
-    $ sudo service influxdb start
-    ```
-* Create the database
-    ```sh
-    $ influx
-    CREATE DATABASE db_meters
-    exit

-    ```
-[*source](https://docs.influxdata.com/influxdb/v1.3/introduction/installation/)
-
-#### Install Grafana*
-
-##### Step-by-step instructions
-* Add APT Repository
-    ```sh
-    $ echo "deb https://dl.bintray.com/fg2it/deb-rpi-1b jessie main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
-    ```
-* Add Bintray key
-    ```sh
-    $ curl https://bintray.com/user/downloadSubjectPublicKey?username=bintray | sudo apt-key add -
-    ```
-* Now install
-    ```sh
-    $ sudo apt-get update && sudo apt-get install grafana

-    ```
-* Start the service using systemd:
-    ```sh
-    $ sudo systemctl daemon-reload
-    $ sudo systemctl start grafana-server
-    $ systemctl status grafana-server
-    ```
-* Enable the systemd service so that Grafana starts at boot.
-    ```sh
-    $ sudo systemctl enable grafana-server.service
-    ```
-* Go to http://localhost:3000 and login using admin / admin (remember to change password)
-[*source](http://docs.grafana.org/installation/debian/)
-
-#### Install Energy Meter Logger:
-* Download and install from Github
-    ```sh
-    $ git clone https://github.com/samuelphy/energy-meter-logger
-    ```
-* Run setup script (must be executed as root (sudo) if the application needs to be started from rc.local, see below)
-    ```sh
-    $ cd energy-meter-logger
-    $ sudo python setup.py install
-    ```    
-* Make script file executable
-    ```sh
-    $ chmod 777 read_energy_meter.py
-    ```
-* Edit meters.yml to match your configuration
-* Test the configuration by running:
-    ```sh
-    ./read_energy_meter.py
-    ./read_energy_meter.py --help # Shows you all available parameters
-    ```
-* To run the python script at system startup. Add to following lines to the end of /etc/rc.local but before exit:
-    ```sh
-    # Start Energy Meter Logger
-    /home/pi/energy-meter-logger/read_energy_meter.py --interval 60 > /var/log/energy_meter.log &
-    ```
-    Log with potential errors are found in /var/log/energy_meter.log
+The original project has been documented at [Hackster](https://www.hackster.io/samuelphy/energy-meter-logger-6a3468). Please follow the instructions there for more detailed information. 
+Also check original project on [Github](https://github.com/samuelphy/energy-meter-logger).

SDM120.yml

@@ -1,10 +0,0 @@
-Voltage Phase 1 : 0
-Current Phase 1 : 6
-Active power Phase 1 : 12
-Apparent power Phase 1 : 18
-Reactive power Phase 1 : 24
-Power factor Phase 1 : 30
-Frequency : 70
-Import active energy : 72
-Export active energy : 74
-Total active energy : 86

SDM630.yml

@@ -1,23 +0,0 @@
-Voltage Phase 1 : 0
-Voltage Phase 2 : 2
-Voltage Phase 3 : 4
-Current Phase 1 : 6
-Current Phase 2 : 8
-Current Phase 3 : 10
-Active power Phase 1 : 12
-Active power Phase 2 : 14
-Active power Phase 3 : 16
-Apparent power Phase 1 : 18
-Apparent power Phase 2 : 20
-Apparent power Phase 3 : 22
-Reactive power Phase 1 : 24
-Reactive power Phase 2 : 26
-Reactive power Phase 3 : 28
-Power factor Phase 1 : 30
-Power factor Phase 2 : 32
-Power factor Phase 3 : 34
-Sum of line currents : 48
-Frequency : 70
-Import active energy : 72
-Export active energy : 74
-Total active energy : 86

influx_config.yml

@@ -1,5 +0,0 @@
-host : 'localhost'
-port : 8086
-user : 'root'
-password : 'root'
-dbname : 'db_meters'

meters.yml

@@ -1,17 +1,26 @@
 meters:
-    - name : Meter Group 1
-      type : SDM120.yml
-      id : 1     # this is the slave address number
-      baudrate : 9600   # Baud
-      bytesize : 8
-      parity : even # none | odd | even
-      stopbits : 1
-      timeout  : 0.5   # seconds
-    - name : Meter Group 2
-      type : SDM630.yml
-      id : 13     # this is the slave address number
-      baudrate : 9600   # Baud
-      bytesize : 8
-      parity : none # none | odd | even
-      stopbits : 1
-      timeout  : 0.5   # seconds
+  - id : 1     # slave address number
+    name : Meter-1
+    type : SDM120
+    baudrate : 9600
+    bytesize : 8
+    parity : even # none | odd | even
+    stopbits : 1
+    timeout  : 0.3   # seconds
+  - id : 2     # slave address number
+    name : Meter-2
+    type : SDM120
+    baudrate : 9600
+    bytesize : 8
+    parity : even # none | odd | even
+    stopbits : 1
+    timeout  : 0.3   # seconds
+  - id: 3     # slave address number
+    name: Meter-3
+    type: SDM120
+    baudrate : 9600
+    bytesize : 8
+    parity : even # none | odd | even
+    stopbits : 1
+    timeout  : 0.3   # seconds
+  

metertype_SDM120.yml

@@ -0,0 +1,40 @@
+momentary:
+  #Voltage__V: 
+  #  address: 0x00
+  #  decimals: 0
+  Current__A: 
+    address: 0x06
+    decimals: 2
+  Power_Active__W: 
+    address: 0x0C
+    decimals: 0
+  Power_Apparent__VA: 
+    address: 0x12
+    decimals: 0
+  #Power_Reactive__VAr: 
+  #  address: 0x18
+  #  decimals: 0
+  PowerFactor: 
+    address: 0x1E
+    decimals: 2
+  #Frequency__Hz: 0x46
+energy:
+  #Energy_Active_Import__kWh: 
+  # address: 0x48
+  # decimals: 3
+  #Energy_Active_Export__kWh: 
+  #  address: 0x4A
+  #  decimals: 3
+  #Energy_Reactive_Import__kvarh: 
+  #  address: 0x4C
+  #  decimals: 3
+  #Energy_Reactive_Export__kvarh: 
+  #  address: 0x4E
+  #  decimals: 3
+  Energy_Total__kWh: 
+    address: 0x0156
+    decimals: 3
+  Energy_Reactive_Total__kvarh: 
+    address: 0x0158
+    decimals: 3
+    

metertype_SDM630.yml

@@ -0,0 +1,80 @@
+momentary:
+  #Voltage_P1__V: 
+  #  address: 0x00
+  #  decimals: 0
+  #Voltage_P2__V: 
+  #  address: 0x02
+  #  decimals: 0
+  #Voltage_P3__V: 
+  #  address: 0x04
+  #  decimals: 0
+  Current_P1__A: 
+    address: 0x06
+    decimals: 2
+  Current_P2__A: 
+    address: 0x08
+    decimals: 2
+  Current_P3__A: 
+    address: 0x0A
+    decimals: 2
+  Power_Active_P1__W: 
+    address: 0x0C
+    decimals: 0
+  Power_Active_P2__W: 
+    address: 0x0E
+    decimals: 0
+  Power_Active_P3__W: 
+    address: 0x10
+    decimals: 0
+  Power_Apparent_P1__VA: 
+    address: 0x12
+    decimals: 0
+  Power_Apparent_P2__VA: 
+    address: 0x14
+    decimals: 0
+  Power_Apparent_P3__VA: 
+    address: 0x16
+    decimals: 0
+  Power_Reactive_P1__VAr: 
+    address: 0x18
+    decimals: 0
+  Power_Reactive_P2__VAr: 
+    address: 0x1A
+    decimals: 0
+  Power_Reactive_P3__VAr: 
+    address: 0x1C
+    decimals: 0
+  #PowerFactor_P1: 
+  #  address: 0x1E
+  #  decimals: 2
+  #PowerFactor_P2: 
+  #  address: 0x20
+  #  decimals: 2
+  #PowerFactor_P3: 
+  #  address: 0x22
+  #  decimals: 2
+  Currents_Sum__A:
+    address: 0x30
+    decimals: 2
+  #Frequency__Hz: 
+  #  address: 0x46
+  #  decimals: 2
+energy:
+  #Energy_Active_Import__kWh: 
+  #  address: 0x48
+  #  decimals: 3
+  #Energy_Active_Export__kWh: 
+  #  address: 0x4A
+  #  decimals: 3
+  #Energy_Reactive_Import__kvarh: 
+  #  address: 0x4C
+  #  decimals: 3
+  #Energy_Reactive_Export__kvarh: 
+  #  address: 0x4E
+  #  decimals: 3
+  Energy_Total__kWh: 
+    address: 0x156
+    decimals: 3
+  Energy_Reactive_Total__kvarh: 
+    address: 0x0158
+    decimals: 3

modbuslog.ini

@@ -0,0 +1,102 @@
+[main]
+publish_on_mqtt = True
+store_in_influxdb = True
+
+[rs485]
+serialdevice = /dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK05DZIG-if00-port0
+serialtimeout = 0.3
+read_retries = 4
+
+# stops script execution, so use with caution!
+raise_error_on_reading_failure = False 
+
+# Sleep for some time between readings of one meter avoid errors
+# i got best results (nearly no retries) with sleep_between_instruments = 0.03 and sleep_between_readings = 0 
+# when reading 3 SDM120 meters set (9600 baud, parity EVEN)
+# i.e. reading current, power, appearent power and power factor from 1 meter needs about 300ms 
+# providing an update rate of more than 1/s for 3 meters
+sleep_between_readings = 0
+sleep_between_instruments = 0.03
+report_instrument_read_retries = False
+
+# publish ongoing instrument read errors on MQTT topic - MQTT must be enabled and error topic set in mqtt section
+# time in s after that repeated instrument reading errors are published via MQTT
+# interval in s to publish repeated instrument reading errors via MQTT
+readingerror_publish_after = 60
+readingerror_publish_interval = 300
+
+[mqtt]
+enable = true
+server = mqtt.lan
+port = 1883
+user = 
+password = 
+
+# topic prefix
+topic_prefix = PowerMeters/Top5
+
+# topic for error messages
+topic_error = PowerMeters/Top5/Modbus/ERROR
+
+[filelog]
+# needed for calculating today/yesterday totals, so only disable if that is not needed
+enable = True
+storage_path = /home/pi/modbuslog
+
+[meters]
+# s, base interval for reading instruments
+# set to 0 to acquire data as fast as possible
+interval_momentary = 0
+
+# s, interval for reporting momentary readings, 0 to report immediately
+# (overruled by powerdelta settings)
+interval_report_momentary = 60
+
+# s, interval for reporting kWh-readings
+# this runs within the base meter reading method, but is performed less often than "interval_momentary" 
+# by measuring elapsed time since last reading, so actual interval can vary, especially when
+# high "interval_momentary" is set to a higher value. To avoid that set 
+# "interval_momentary" (= command parameter --interval) to desired value 
+# and configure "meters_use_only_one_interval" to True if momentary data should not be read more often
+interval_energy = 60
+
+# use only interval 1 (which is interval_momentary)
+use_only_one_interval = False
+
+# report momentary readings immediately on sudden power changes
+# this uses the readings in category "power" in readings_names.yml
+# value is in % (decimal notation)
+report_on_powerdelta_enable = true
+report_on_powerdelta_low = 0.95
+report_on_powerdelta_high = 1.05
+# different powerdelta configuration for low load conditions
+report_on_lowpower_powerdelta_low = 0.70
+report_on_lowpower_powerdelta_high = 1.30
+# define max Watts for low power tresholds
+report_on_lowpower_treshold = 10
+
+# add reading time to MQTT messages and InfluxDB
+# this is the measured time in seconds that was needed to get all the data from the instrument
+send_readtime = False
+
+[readings]
+# default decimal places for data conversion
+# used for readings that don´t have this setting configured separately in meters_types.yml
+default_decimals = 3
+
+[influxdb]
+write_energy_today_total = True
+write_energy_yesterday_total = True
+separate_momentary_database = True
+host = localhost
+port = 8086
+user = 
+password = 
+database = energymeters
+ 
+[influxdb_momentary]
+host = localhost
+port = 8086
+user = 
+password = 
+database = energy_momentary

modbuslog.py

@@ -0,0 +1,721 @@
+#!/usr/bin/env python3
+
+from influxdb import InfluxDBClient
+from os import path
+import configparser
+import sys
+import os
+import minimalmodbus
+import time
+import datetime
+import yaml
+import logging
+import json
+import paho.mqtt.client as mqtt
+
+# Change working dir to the same dir as this script
+os.chdir(sys.path[0])
+
+config = configparser.ConfigParser()
+config.read('modbuslog.ini')
+#print(config.sections())
+
+# additional conffile names
+conffile_meter_types = 'meter_types.yml'
+conffile_readings_names = 'readings_names.yml'
+
+
+# config vars used more than once or updateable via commandline argument are stored as global vars
+conf_modbus_read_retries = config['rs485'].getint('read_retries', 4)
+conf_modbus_raise_error_on_reading_failure = config['rs485'].getboolean('raise_error_on_reading_failure', False)
+conf_modbus_sleep_between_readings = config['rs485'].getfloat('sleep_between_readings', 0.1)
+conf_modbus_sleep_between_instruments = config['rs485'].getfloat('sleep_between_instruments', 0.7)
+
+conf_publish_on_mqtt = config['main'].getboolean('publish_on_mqtt', False)
+conf_store_in_influxdb = config['main'].getboolean('store_in_influxdb', False)
+
+conf_mqtt_enabled = config['mqtt'].getboolean('enable', False)
+
+conf_mqtt_topic_prefix = config['mqtt'].get('topic_prefix') #must NOT end with / !!
+if conf_mqtt_topic_prefix[-1:] == '/':
+    conf_mqtt_topic_prefix = conf_mqtt_topic_prefix[0:-1]
+    
+conf_mqtt_topic_error = config['mqtt'].get('topic_error') #must NOT end with / !!
+
+conf_storage_path = config['filelog'].get('storage_path')
+if conf_storage_path[-1:] != '/':
+    conf_storage_path += '/'
+    
+meters_interval_momentary = config['meters'].getint('interval_momentary', 1) # s - base interval for reading instruments 
+meters_interval_report_momentary = config['meters'].getint('interval_report_momentary', 60)    # interval for reporting momentary readings, 0 to report immediately, overruled by powerdelta settings
+meters_interval_energy = config['meters'].getint('interval_energy', 60)    # s - interval for reporting kWh-readings
+                            # for now this is not a seperate function but based on "meters_interval_momentary" 
+                            # easuring elapsed time since last reading, so actual interval can vary, especially when
+                            # high "meters_interval_momentary" is set. To avoid that set 
+                            # "meters_interval_momentary" (= command parameter --interval) to desired value 
+                            # and configure "meters_use_only_one_interval" to True
+meters_use_only_one_interval = config['meters'].getboolean('use_only_one_interval', False)   # use only interval 1 "meters_interval_momentary"
+meters_report_on_powerdelta_low  = config['meters'].getfloat('report_on_powerdelta_low', 0.95)  # % in decimal notation - immediately report if power value changes by more then this %
+meters_report_on_powerdelta_high = config['meters'].getfloat('report_on_powerdelta_high', 1.05)      # % in decimal notation - immediately report if power value changes by more then this %
+meters_report_on_lowpower_treshold = config['meters'].getint('report_on_lowpower_treshold', 10) # treshold under which measured power is considered "low" and different powerdeltas are used
+meters_report_on_lowpower_powerdelta_low  = config['meters'].getfloat('report_on_lowpower_powerdelta_low', 0.70)   # % in decimal notation - immediately report if power value changes by more then this %
+meters_report_on_lowpower_powerdelta_high = config['meters'].getfloat('report_on_lowpower_powerdelta_high', 1.30)   # % in decimal notation - immediately report if power value changes by more then this %
+##report_instrument_read_retries = config['rs485'].getboolean('report_instrument_read_retries', False)
+
+conf_send_meters_readTime = config['meters'].getboolean('send_readtime', True)
+conf_default_decimals = config['readings'].getint('default_decimals', 3)
+
+
+conf_readingerror_publish_after = config['rs485'].getint('readingerror_publish_after', 60)         # time in s after that repeated instrument reading errors are published via MQTT
+conf_readingerror_publish_interval = config['rs485'].getint('readingerror_publish_interval', 300)     # interval in s to publish repeated instrument reading errors via MQTT
+
+
+# -------------------------------------------------------------
+
+# global variables - not for configuration
+args_output_verbose1 = False
+args_output_verbose2 = False
+
+
+influxdb_write_energy_today_total = True
+influxdb_write_energy_yesterday_total = True
+
+
+
+
+
+
+class DataCollector:
+    def __init__(self, influx_client_momentary, influx_client_energy, meter_yaml):
+        self.influx_client_momentary = influx_client_momentary
+        self.influx_client_energy = influx_client_energy
+        self.meter_yaml = meter_yaml
+        self.max_iterations = None  # run indefinitely by default
+        #self.meter_types = None
+        self.meter_types = dict()
+        self.meter_types_last_change = dict()
+        #self.meter_types_last_change = -1
+        self.meter_map = None
+        self.meter_map_last_change = -1
+        self.meter_configuration_lastchecktime = None
+        self.meter_typesconfiguration_lastchecktime = None
+        self.lastMomentaryReportTime = dict()
+        self.lastEnergyUpdate = dict()
+        self.lastReadingErrorTime = dict()
+        self.lastReadingErrorPublishtime = dict()
+        #self.totalEnergy = dict()
+        self.saved_energy_today_min = dict()
+        self.data_momentary_last = dict()
+        self.saved_energy_yesterday_total = dict()     # remember total energy for each meter 
+        self.saved_todays_date = dict()      # remember today´s date for each meter, needed to check for date rollover in order to calculate energy yesterday/today
+        log.info('Meters:')
+        #for meter in sorted(self.get_meters()):   # does not work in Python 3, so dont sort for now
+        
+        # reading conffile_readings_names
+        self.readingsNames = yaml.load(open(conffile_readings_names), Loader=yaml.FullLoader)
+        
+        for meter in self.get_meters():
+            log.info('\t {} <--> {}'.format( meter['id'], meter['name']))
+    
+    def load_meter_type(self, metertype):
+        log.info("Loading meter type: " + metertype)
+        conffile_meter_type = "metertype_" + metertype + ".yml"
+        assert path.exists(conffile_meter_type), 'Meters configuration not found: %s' % conffile_meter_type
+        lastchange = self.meter_types_last_change.get(metertype, None)
+        lastchange_file = path.getmtime(conffile_meter_type)
+        if lastchange == None or (lastchange and lastchange_file != lastchange):
+            try:
+                log.info('Reloading meter type configuration for ' + metertype + 'as file changed')
+                self.meter_types[metertype] = yaml.load(open(conffile_meter_type), Loader=yaml.FullLoader)
+                self.meter_types_last_change[metertype] = lastchange_file
+                log.debug('Reloaded meters configuration')
+            except Exception as e:
+                log.warning('Failed to re-load meter type configuration, going on with the old one.')
+                log.warning(e)
+    
+    def check_load_reload_meter_types(self):
+        log.debug("")
+        log.debug("checking loaded meter types...")
+        for metertype in self.meter_types: 
+            log.debug(metertype)
+            self.load_meter_type(metertype)
+        log.debug("")
+        
+    def get_meters(self):
+        reloadconf = False
+        ts = int(time.time())
+        if self.meter_configuration_lastchecktime == None or (ts - self.meter_configuration_lastchecktime) > 60:
+            self.meter_configuration_lastchecktime = ts
+            assert path.exists(self.meter_yaml), 'Meter map not found: %s' % self.meter_yaml
+            if path.getmtime(self.meter_yaml) != self.meter_map_last_change:
+                reloadconf = True
+        if reloadconf:
+            try:
+                log.info('Reloading meter map as file changed')
+                new_map = yaml.load(open(self.meter_yaml), Loader=yaml.FullLoader)
+                self.meter_map = new_map['meters']
+                self.meter_map_last_change = path.getmtime(self.meter_yaml)                
+                log.debug('Reloaded meter map')
+                for entry in self.meter_map:
+                    log.debug(entry['type'])
+                    self.load_meter_type(entry['type'])
+
+            except Exception as e:
+                log.warning('Failed to re-load meter map, going on with the old one.')
+                log.warning(e)
+                
+        if self.meter_typesconfiguration_lastchecktime == None or (ts - self.meter_typesconfiguration_lastchecktime) > 60:
+            self.meter_typesconfiguration_lastchecktime = ts
+            self.check_load_reload_meter_types()
+            
+        return self.meter_map
+
+    def collect_and_store(self):
+        #instrument.debug = True
+        meters = self.get_meters()
+        
+        instrument = minimalmodbus.Instrument(config['rs485'].get('serialdevice','/dev/ttyUSB0'), config['rs485'].getfloat('serialtimeout', 1.0))
+        instrument.mode = minimalmodbus.MODE_RTU   # rtu or ascii mode
+        
+        data_momentary = dict()
+        data_energy = dict()
+        meter_id_name = dict() # mapping id to name
+
+        for meter in meters:
+            meterReadingError_momentary = False
+            meterReadingError_energy = False
+            if conf_modbus_sleep_between_instruments  > 0:
+                time.sleep(conf_modbus_sleep_between_instruments)
+            meter_id_name[meter['id']] = meter['name']
+            instrument.serial.baudrate = meter['baudrate']
+            instrument.serial.bytesize = meter['bytesize']
+            if meter['parity'] == 'none':
+                instrument.serial.parity = minimalmodbus.serial.PARITY_NONE
+            elif meter['parity'] == 'odd':
+                instrument.serial.parity = minimalmodbus.serial.PARITY_ODD
+            elif meter['parity'] == 'even':
+                instrument.serial.parity = minimalmodbus.serial.PARITY_EVEN
+            else:
+                log.error('No parity specified')
+                raise
+            instrument.serial.stopbits = meter['stopbits']
+            instrument.serial.timeout  = meter['timeout']    # seconds
+            instrument.address = meter['id']    # this is the slave address number
+
+            log.debug('\nReading meter %s \'%s\'' % (meter['id'], meter_id_name[meter['id']]))
+            start_time = time.time()
+            
+            #if not self.meter_types.get(meter['type'], False):
+            #    self.load_meter_type(meter['type'])
+                
+            readings = self.meter_types[meter['type']]
+            if args_output_verbose2:
+                log.debug("")
+                log.debug("Meter Type " + meter['type'] + " - defined readings:")
+                log.debug(json.dumps(readings, indent = 4))
+                log.debug("")
+            
+            data_momentary[meter['id']] = dict()
+            data_energy[meter['id']] = dict()
+
+            reading_success_momentary = 0
+            for reading in readings['momentary']:
+                # to prevent random readout errors, e.g. CRC check fail, sleep for a short time between the readings
+                if conf_modbus_sleep_between_readings > 0:
+                    time.sleep(conf_modbus_sleep_between_readings) # Sleep between readings to avoid read errors
+                retries = conf_modbus_read_retries
+                
+                # get decimals needed from meter_types config
+                decimals = readings['momentary'][reading].get('decimals', conf_default_decimals)
+                
+                while retries > 0:
+                    try:
+                        retries -= 1
+                        data_momentary[meter['id']][reading] = round(instrument.read_float(readings['momentary'][reading]['address'], 4, 2), decimals)
+                        log.debug('OK read meter {}, {} retries => \'{}\' = \'{}\''.format(meter['id'], conf_modbus_read_retries - retries, reading, data_momentary[meter['id']][reading]))
+                        retries = 0
+                        reading_success_momentary += 1
+                        pass
+                    except ValueError as ve:
+                        log.warning('Value Error while reading register {} from meter {}. Retries left {}.'
+                               .format(readings['momentary'][reading]['address'], meter['id'], retries))
+                        log.error(ve)
+                        if retries == 0 and conf_modbus_raise_error_on_reading_failure:
+                            raise RuntimeError
+                    except TypeError as te:
+                        log.warning('Type Error while reading register {} from meter {}. Retries left {}.'
+                               .format(readings['momentary'][reading]['address'], meter['id'], retries))
+                        log.error(te)
+                        if retries == 0 and conf_modbus_raise_error_on_reading_failure:
+                            raise RuntimeError
+                    except IOError as ie:
+                        log.warning('IO Error while reading register {} from meter {}. Retries left {}.'
+                               .format(readings['momentary'][reading]['address'], meter['id'], retries))
+                        log.error(ie)
+                        if retries == 0 and conf_modbus_raise_error_on_reading_failure:
+                            raise RuntimeError
+                    except:
+                        log.error("Unexpected error:", sys.exc_info()[0])
+                        raise
+            if reading_success_momentary < len(readings['momentary']):
+                log.debug("THERE WERE READING ERRORS")
+                meterReadingError_momentary = True
+            
+            # report momentary interval
+            reportMomentary = False
+            if meters_interval_report_momentary > 0:
+                ts = int(time.time())
+                lastMomentaryReportTime = self.lastMomentaryReportTime.get(meter['id'], False)
+                if lastMomentaryReportTime:
+                    tdiff = ts - lastMomentaryReportTime
+                    if (tdiff > meters_interval_report_momentary):
+                        log.debug('Reporting momentary readings for meter %s' % meter['id'])
+                        reportMomentary = True
+                        self.lastMomentaryReportTime[meter['id']] = ts
+                else:
+                    log.debug('No lastMomentaryReportTime has yet been saved for meter %s' % meter['id'])
+                    reportMomentary = True
+                    self.lastMomentaryReportTime[meter['id']] = ts
+            else:
+                reportMomentary = True
+            
+             
+            # override meters_interval_report_momentary if power has changed for more than configured powerdelta and no interval reporting is due in this iteration
+            if config['meters'].getboolean('report_on_powerdelta_enable', False) and not reportMomentary:
+                lastValues = self.data_momentary_last.get(meter['id'], None)
+                                
+                for usedReading in data_momentary[meter['id']].keys():
+                    currentValue = data_momentary[meter['id']][usedReading]
+                    for powerreadingname in self.readingsNames['power']:
+                        if usedReading == powerreadingname:
+                            lastValue = None
+                            if lastValues != None:
+                                lastValue = lastValues.get(powerreadingname, None)
+                            if lastValue != None:
+                                if (currentValue >= meters_report_on_lowpower_treshold):
+                                    powerdelta_high = meters_report_on_powerdelta_high
+                                    powerdelta_low = meters_report_on_powerdelta_low
+                                else:
+                                    powerdelta_high = meters_report_on_lowpower_powerdelta_high
+                                    powerdelta_low = meters_report_on_lowpower_powerdelta_low
+                                    
+                                if (currentValue > (lastValue * powerdelta_high)):
+                                    log.debug(powerreadingname + " INCREASED by more than factor " + str(powerdelta_high) + "     currentValue=" + str(currentValue) + "  lastValue=" + str(lastValue))
+                                    reportMomentary = True
+                                if (currentValue < (lastValue * powerdelta_low)):
+                                    log.debug(powerreadingname + " DECREASED by more than factor " + str(powerdelta_low) + "     currentValue=" + str(currentValue) + "  lastValue=" + str(lastValue))
+                                    reportMomentary = True
+                            if lastValues == None:
+                                self.data_momentary_last[meter['id']] = dict()
+                            self.data_momentary_last[meter['id']][powerreadingname] = data_momentary[meter['id']][powerreadingname]
+                    
+            
+            # influxdb
+            t_utc = datetime.datetime.utcnow()
+            t_str = t_utc.isoformat() + 'Z'
+            
+            if conf_store_in_influxdb and not meterReadingError_momentary and reportMomentary:
+                jsondata_momentary = [
+                    {
+                        'measurement': 'energy',
+                        'tags': {
+                            'meter': meter_id_name[meter['id']],
+                        },
+                        'time': t_str,
+                        'fields': data_momentary[meter['id']]
+                    }
+                ]
+                if args_output_verbose1:
+                    print(json.dumps(jsondata_momentary, indent = 4))
+                try:
+                    self.influx_client_momentary.write_points(jsondata_momentary)
+                except Exception as e:
+                    log.error('Data not written!')
+                    log.error(e)
+                
+            if conf_send_meters_readTime:
+                readtime = round(time.time() - start_time, 3)
+                log.debug("Read time: " + str(readtime))
+                data_momentary[meter['id']]['Read time'] = readtime
+                if conf_mqtt_enabled and conf_publish_on_mqtt:
+                        mqttc.publish(conf_mqtt_topic_prefix + "/" + meter_id_name[meter['id']] + "/ReadTime", str(readtime))
+            
+            if conf_mqtt_enabled and conf_publish_on_mqtt and reportMomentary:
+                for reading in readings['momentary']:
+                    tmpreading = data_momentary[meter['id']].get(reading, None)
+                    if tmpreading != None:
+                        if tmpreading.is_integer():
+                            tmpreading = int(tmpreading)
+                        #mqttc.publish(conf_mqtt_topic_prefix + "/" + meter_id_name[meter['id']] + "/" + reading, str('{0:.3f}'.format(tmpreading)))
+                        log.debug("MQTT pub: '"+conf_mqtt_topic_prefix + "/" + meter_id_name[meter['id']] + "/" + reading + "' = '" + str(tmpreading) + "'")
+                        mqttc.publish(conf_mqtt_topic_prefix + "/" + meter_id_name[meter['id']] + "/" + reading, str(tmpreading))
+            
+            
+            
+            if meters_use_only_one_interval:
+                readEnergyData = True
+            else:
+                readEnergyData = False
+                ts = int(time.time())
+                lastUpdate = self.lastEnergyUpdate.get(meter['id'], False)
+                if lastUpdate:
+                    tdiff = ts - lastUpdate
+                    if (tdiff > meters_interval_energy):
+                        readEnergyData = True
+                        self.lastEnergyUpdate[meter['id']] = ts
+                else:
+                    log.debug('No lastEnergyUpdate has yet been saved for meter %s' % meter['id'])
+                    readEnergyData = True
+                    self.lastEnergyUpdate[meter['id']] = ts
+            
+            
+            
+            # save and restore yesterday´s total energy to calculate today´s energy
+            # check if total energy from yesterday is stored in memory, if not try to get it from saved file
+            today = datetime.date.today()
+            today_str = today.strftime('%Y%m%d')
+            yesterday = today - datetime.timedelta(days = 1)
+            yesterday_str = yesterday.strftime('%Y%m%d')
+            
+            # check for date rollover
+            dateRollover = False
+            savedtoday = self.saved_todays_date.get(meter['id'], False)
+            if not savedtoday or savedtoday != today:
+                log.debug("date rollover happened or no date has been saved yet for meter " + str(meter['id']))
+                if savedtoday and savedtoday == yesterday:
+                    # a date rollover just happened, so change todays date to current and proceed with what has to be done
+                    dateRollover = True
+                    readEnergyData = True
+                    #log.debug(savedtoday)
+                self.saved_todays_date[meter['id']] = today
+                
+                
+            if readEnergyData:
+                reading_success_energy = 0
+                for reading in readings['energy']:
+                    # to prevent random readout errors, e.g. CRC check fail, sleep for a short time between the readings
+                    if conf_modbus_sleep_between_readings > 0:
+                        time.sleep(conf_modbus_sleep_between_readings) # Sleep between readings to avoid read errors
+                    retries = conf_modbus_read_retries
+                    
+                    # get decimals needed from meter_types config
+                    decimals = readings['energy'][reading].get('decimals', conf_default_decimals)
+                
+                    while retries > 0:
+                        try:
+                            retries -= 1
+                            data_energy[meter['id']][reading] = round(instrument.read_float(readings['energy'][reading]['address'], 4, 2), decimals)
+                            log.debug('OK read meter {}, {} retries => \'{}\' = \'{}\''
+                                .format(meter['id'], conf_modbus_read_retries - retries, reading, data_energy[meter['id']][reading]))
+                            reading_success_energy += 1
+                            retries = 0
+                            pass
+                        except ValueError as ve:
+                            log.warning('Value Error while reading register {} from meter {}. Retries left {}.'
+                                .format(readings['energy'][reading]['address'], meter['id'], retries))
+                            log.error(ve)
+                            if retries == 0 and conf_modbus_raise_error_on_reading_failure:
+                                raise RuntimeError
+                        except TypeError as te:
+                            log.warning('Type Error while reading register {} from meter {}. Retries left {}.'
+                                .format(readings['energy'][reading]['address'], meter['id'], retries))
+                            log.error(te)
+                            if retries == 0 and conf_modbus_raise_error_on_reading_failure:
+                                raise RuntimeError
+                        except IOError as ie:
+                            log.warning('IO Error while reading register {} from meter {}. Retries left {}.'
+                                .format(readings['energy'][reading]['address'], meter['id'], retries))
+                            log.error(ie)
+                            if retries == 0 and conf_modbus_raise_error_on_reading_failure:
+                                raise RuntimeError
+                        except:
+                            log.error("Unexpected error:", sys.exc_info()[0])
+                            if conf_modbus_raise_error_on_reading_failure:
+                                raise
+                if reading_success_energy < len(readings['energy']):
+                    log.debug("THERE WERE READING ERRORS")
+                    meterReadingError_energy = True
+                    
+                
+                file_path_meter = conf_storage_path + meter_id_name[meter['id']] + "/"
+                file_today_min = file_path_meter + today_str + "_min.txt"
+                file_yesterday_total = file_path_meter + yesterday_str + "_total.txt"
+                
+                energy_today_total = 0
+                energy_yesterday_min = 0
+                energy_today_min = self.saved_energy_today_min.get(meter['id'], None)
+                
+                if dateRollover:
+                    energy_today_min = None
+                if energy_today_min == None:
+                    exists = os.path.isfile(file_today_min)
+                    if exists:
+                        # load energy_today_min from file if exists
+                        f = open(file_today_min, "r")
+                        if f.mode == 'r':
+                            contents = f.read()
+                            f.close()
+                        energy_today_min = float(contents)
+                        self.saved_energy_today_min[meter['id']] = energy_today_min
+                        log.debug(meter_id_name[meter['id']] + " - Energy Today min read from file -> = " + str(energy_today_min) + " kWh")
+                    else:
+                        # save current Energy_total to min-file
+                        if not os.path.exists(file_path_meter):
+                            os.mkdir(file_path_meter)
+                        f = open(file_today_min, "w+")
+                        energy_today_min = data_energy[meter['id']][self.readingsNames['energy_total']]
+                        self.saved_energy_today_min[meter['id']] = energy_today_min
+                        f.write(str('{0:.3f}'.format(energy_today_min)))
+                        f.close()
+                log.debug(meter_id_name[meter['id']] + " - Energy Today Min: " + str(energy_today_min) + " kWh")
+                
+                try:
+                    energy_today_total = data_energy[meter['id']][self.readingsNames['energy_total']] - energy_today_min
+                    log.debug(meter_id_name[meter['id']] + " - Energy Today total: " + str('{0:.3f}'.format(energy_today_total)) + " kWh")
+                except:
+                    pass
+                
+                
+                
+                energy_yesterday_total = self.saved_energy_yesterday_total.get(meter['id'], None)
+                if dateRollover:
+                    energy_yesterday_total = None
+                if energy_yesterday_total == None:
+                    exists = os.path.isfile(file_yesterday_total)
+                    if exists:
+                        # load energy_yesterday_total from file if exists
+                        f = open(file_yesterday_total, "r")
+                        if f.mode == 'r':
+                            contents = f.read()
+                            f.close()
+                        energy_yesterday_total = float(contents)
+                        self.saved_energy_yesterday_total[meter['id']] = energy_yesterday_total
+                        log.debug(meter_id_name[meter['id']] + " - Energy Yesterday total read from file -> = " + str(energy_yesterday_total) + " kWh")
+                    else:
+                        file_yesterday_min = file_path_meter + yesterday_str + "_min.txt"
+                        exists = os.path.isfile(file_yesterday_min)
+                        if exists:
+                            # load yesterday_min from file
+                            #if args_output_verbose1:
+                            #    print("file file_yesterday_min exists")
+                            f = open(file_yesterday_min, "r")
+                            if f.mode == 'r':
+                                contents =f.read()
+                                f.close()
+                            energy_yesterday_min = float(contents)
+                            log.debug(meter_id_name[meter['id']] + " - Energy yesterday min: " + str(energy_yesterday_min) + " kWh")
+                            
+                            energy_yesterday_total = round(energy_today_min - energy_yesterday_min, 3)
+                            ###log.debug(meter_id_name[meter['id']] + " - Energy yesterday total: " + str(energy_yesterday_total))
+                            
+                            if not os.path.exists(file_path_meter):
+                                os.mkdir(file_path_meter)
+                            f = open(file_yesterday_total, "w+")
+                            f.write(str('{0:.3f}'.format(energy_yesterday_total)))
+                            f.close()
+                        #else:
+                        #    # file yesterday_min does not exist
+                log.debug(meter_id_name[meter['id']] + " - Energy Yesterday Total: " + str(energy_yesterday_total) + " kWh")
+                
+                if influxdb_write_energy_today_total:
+                    data_energy[meter['id']][self.readingsNames['energy_today']] = energy_today_total
+                if influxdb_write_energy_yesterday_total:
+                    data_energy[meter['id']][self.readingsNames['energy_yesterday']] = energy_yesterday_total
+                
+                t_utc = datetime.datetime.utcnow()
+                t_str = t_utc.isoformat() + 'Z'
+                
+                if conf_store_in_influxdb and not meterReadingError_energy:
+                    jsondata_energy = [
+                        {
+                            'measurement': 'energy',
+                            'tags': {
+                                'meter': meter_id_name[meter['id']],
+                            },
+                            'time': t_str,
+                            'fields': data_energy[meter['id']]
+                        }
+                    ]
+                    if args_output_verbose1:
+                        print(json.dumps(jsondata_energy, indent = 4))
+                    try:
+                        self.influx_client_energy.write_points(jsondata_energy)
+                    except Exception as e:
+                        log.error('Data not written!')
+                        log.error(e)
+                
+                if conf_send_meters_readTime:
+                    readtime = round(time.time() - start_time, 3)
+                    log.debug("Read time: " + str(readtime))
+                    data_energy[meter['id']]['Read time'] = readtime
+                    if conf_mqtt_enabled and conf_publish_on_mqtt:
+                        mqttc.publish(conf_mqtt_topic_prefix + "/" + meter_id_name[meter['id']] + "/ReadTime", str(readtime))
+                
+                if conf_mqtt_enabled and conf_publish_on_mqtt:
+                    for reading in readings['energy']:
+                        tmpreading = data_energy[meter['id']].get(reading, None)
+                        if tmpreading != None:
+                            if tmpreading.is_integer():
+                                tmpreading = int(tmpreading)
+                            #mqttc.publish(conf_mqtt_topic_prefix + "/" + meter_id_name[meter['id']] + "/" + reading, str('{0:.3f}'.format(tmpreading)))
+                            log.debug("MQTT pub: '"+conf_mqtt_topic_prefix + "/" + meter_id_name[meter['id']] + "/" + reading + "' = '" + str(tmpreading) + "'")
+                            mqttc.publish(conf_mqtt_topic_prefix + "/" + meter_id_name[meter['id']] + "/" + reading, str(tmpreading))
+                    mqttc.publish(conf_mqtt_topic_prefix + "/" + meter_id_name[meter['id']] + "/" + self.readingsNames['energy_today'], str('{0:.3f}'.format(energy_today_total)))
+                    mqttc.publish(conf_mqtt_topic_prefix + "/" + meter_id_name[meter['id']] + "/" + self.readingsNames['energy_yesterday'], str('{0:.3f}'.format(energy_yesterday_total)))
+            
+            if meterReadingError_momentary or meterReadingError_energy:
+                ts = int(time.time())
+                lasterrortime = self.lastReadingErrorTime.get(meter['id'], 0)
+                if lasterrortime == 0:
+                    self.lastReadingErrorTime[meter['id']] = ts
+                elif (ts - lasterrortime) > conf_readingerror_publish_after:
+                    lasterrorpubtime = self.lastReadingErrorPublishtime.get(meter['id'], 0)
+                    if lasterrorpubtime == 0 or (lasterrorpubtime > 0 and (ts - lasterrorpubtime) > conf_readingerror_publish_interval):
+                        self.lastReadingErrorPublishtime[meter['id']] = ts
+                        if conf_mqtt_enabled and conf_publish_on_mqtt:
+                            lasterrortime_str = datetime.datetime.fromtimestamp(lasterrortime).strftime("%Y-%m-%d %H:%M:%S")
+                            mqttc.publish(conf_mqtt_topic_prefix + "/" + meter_id_name[meter['id']] + "/STATE", "ERROR: could not read MODBUS meter " + meter_id_name[meter['id']] + " with ID=" + str(meter['id']) + " since " + str(lasterrortime_str))
+                            mqttc.publish(conf_mqtt_topic_error, "ERROR: could not read MODBUS meter " + meter_id_name[meter['id']] + " with ID=" + str(meter['id']) + " since " + str(lasterrortime_str))
+            else:
+                self.lastReadingErrorTime[meter['id']] = 0
+                
+# END class DataCollector
+################################
+
+
+
+def mqtt_on_connect(client, userdata, flags, rc):
+    if args_output_verbose1:
+        print("MQTT connected with result code " + str(rc))
+    #client.subscribe("some/topic")
+
+
+def mqtt_on_disconnect(client, userdata, rc):
+    if rc != 0:
+        if print_errors:
+            print("Unexpected MQTT disconnection. Will auto-reconnect")
+
+
+def repeat(interval_sec, max_iter, func, *args, **kwargs):
+    from itertools import count
+    starttime = 0
+    for i in count():
+        if i > 0 and interval_sec > 0:  # do not wait for interval time on first run
+            if ((time.time() - starttime) < interval_sec):
+                sleeptime = interval_sec - (time.time() - starttime)
+                print("\nsleep " + str(sleeptime) + " s")
+                time.sleep(sleeptime)
+        try:
+            starttime = time.time()
+            func(*args, **kwargs)
+        except Exception as ex:
+            log.error(ex)
+        if max_iter and i >= max_iter:
+            return
+
+
+if __name__ == '__main__':
+    import argparse
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--interval', default=meters_interval_momentary,
+                        help='Meter readout interval for momentary values i.e. power, current... - in seconds, default 1s')
+    parser.add_argument('--energyinterval', default=meters_interval_energy,
+                        help='Meter readout interval for energy values, i.e. total kWh - in seconds, default 60s')
+    parser.add_argument('--use-only-one-interval', default=False,
+                        help='Meter readout interval for energy values, i.e. total kWh - in seconds, default 60s', action='store_true')
+    parser.add_argument('--meters', default='meters.yml',
+                        help='YAML file containing Meter ID, name, type etc. Default "meters.yml"')
+    #parser.add_argument('--verbose', '-v', default=0, help='print read data from the instruments to console', action='store_true')
+    parser.add_argument('--verbose', '-v', type=int, default=0, choices=[1, 2], help='print read data from the instruments to console')
+    parser.add_argument('--log', default='CRITICAL',
+                        help='Log levels, DEBUG, INFO, WARNING, ERROR or CRITICAL')
+    parser.add_argument('--logfile', default='',
+                        help='Specify log file, if not specified the log is streamed to console')
+    
+    
+    args = parser.parse_args()
+    
+    loglevel = args.log.upper()
+    logfile = args.logfile
+    
+    # Setup logging
+    log = logging.getLogger('energy-logger')
+    log.setLevel(getattr(logging, loglevel))
+
+    if logfile:
+        loghandle = logging.FileHandler(logfile, 'w')
+        formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
+        loghandle.setFormatter(formatter)
+    else:
+        loghandle = logging.StreamHandler()
+
+    log.addHandler(loghandle)
+    
+    log.info('Started app')
+    
+    #if args.verbose:
+    if int(args.verbose) == 1 or int(args.verbose) == None:
+        args_output_verbose1 = True
+        args_output_verbose2 = False
+        log.info("Verbose Level 1 ON - printing read data to console.")
+    elif int(args.verbose) == 2:
+        args_output_verbose1 = True
+        args_output_verbose2 = True
+        log.info("Verbose Level 2 ON - printing read data and more to console.")
+        
+    interval = int(args.interval)
+    log.info("Interval 1 (for MOMENTARY readings): " + str(interval) + " s")
+    
+    if args.use_only_one_interval:
+        meters_use_only_one_interval = True
+        log.info("Using only Interval 1")
+    else:
+        meters_interval_energy = int(args.energyinterval)
+        log.info("Interval 2 (for ENERGY readings):    " + str(meters_interval_energy) + " s")
+    
+    # create MQTT client object
+    if conf_mqtt_enabled:
+        mqttc = mqtt.Client()
+        mqttc.on_connect = mqtt_on_connect
+        mqttc.on_disconnect = mqtt_on_disconnect
+        ##mqttc.on_message = on_message  # callback for incoming msg (unused)
+        if len(config['mqtt'].get('password')) > 0 or len(config['mqtt'].get('server')) > 0:
+            mqttc.username_pw_set(config['mqtt'].get('user'), config['mqtt'].get('password'))
+        mqttc.connect(config['mqtt'].get('server'), config['mqtt'].getint('port', 1883), 60)
+        mqttc.loop_start()
+        #mqttc.loop_forever()
+
+
+    # Create the InfluxDB object
+    if config['influxdb'].getboolean('separate_momentary_database', False):
+        influxclient_energy = InfluxDBClient(config['influxdb'].get('host'),
+                            config['influxdb'].getint('port', 8086),
+                            config['influxdb'].get('user'),
+                            config['influxdb'].get('password'),
+                            config['influxdb'].get('database'))
+        influxclient_momentary = InfluxDBClient(config['influxdb_momentary'].get('host'),
+                            config['influxdb_momentary'].getint('port', 8086),
+                            config['influxdb_momentary'].get('user'),
+                            config['influxdb_momentary'].get('password'),
+                            config['influxdb_momentary'].get('database'))
+    else:
+        influxclient_energy = InfluxDBClient(config['influxdb'].get('host'),
+                            config['influxdb'].getint('port', 8086),
+                            config['influxdb'].get('user'),
+                            config['influxdb'].get('password'),
+                            config['influxdb'].get('database'))
+        influxclient_momentary = InfluxDBClient(config['influxdb'].get('host'),
+                            config['influxdb'].getint('port', 8086),
+                            config['influxdb'].get('user'),
+                            config['influxdb'].get('password'),
+                            config['influxdb'].get('database'))
+        
+
+    collector = DataCollector(influx_client_momentary=influxclient_momentary,
+                              influx_client_energy=influxclient_energy,
+                              meter_yaml=args.meters)
+                              
+    repeat(interval,
+        max_iter=collector.max_iterations,
+        func=lambda: collector.collect_and_store())

modbuslog.service

@@ -0,0 +1,13 @@
+[Unit]
+Description=ModbusLogger
+StartLimitInterval=0
+
+[Service]
+Type=simple
+Restart=always
+RestartSec=1
+ExecStart=/home/pi/modbuslogger/modbuslog.py
+User=pi
+
+[Install]
+WantedBy=multi-user.target

read_energy_meter.py

@@ -1,194 +0,0 @@
-#!/usr/bin/env python
-
-from influxdb import InfluxDBClient
-from datetime import datetime, timedelta
-from os import path
-import sys
-import os
-import minimalmodbus
-import time
-import yaml
-import logging
-
-# Change working dir to the same dir as this script
-os.chdir(sys.path[0])
-
-class DataCollector:
-    def __init__(self, influx_client, meter_yaml):
-        self.influx_client = influx_client
-        self.meter_yaml = meter_yaml
-        self.max_iterations = None  # run indefinitely by default
-        self.meter_map = None
-        self.meter_map_last_change = -1
-        log.info('Meters:')
-        for meter in sorted(self.get_meters()):
-            log.info('\t {} <--> {}'.format( meter['id'], meter['name']))
-
-    def get_meters(self):
-        assert path.exists(self.meter_yaml), 'Meter map not found: %s' % self.meter_yaml
-        if path.getmtime(self.meter_yaml) != self.meter_map_last_change:
-            try:
-                log.info('Reloading meter map as file changed')
-                new_map = yaml.load(open(self.meter_yaml), Loader=yaml.FullLoader)
-                self.meter_map = new_map['meters']
-                self.meter_map_last_change = path.getmtime(self.meter_yaml)
-            except Exception as e:
-                log.warning('Failed to re-load meter map, going on with the old one.')
-                log.warning(e)
-        return self.meter_map
-
-    def collect_and_store(self):
-        #instrument.debug = True
-        meters = self.get_meters()
-        t_utc = datetime.utcnow()
-        t_str = t_utc.isoformat() + 'Z'
-
-        instrument = minimalmodbus.Instrument('/dev/ttyAMA0', 1) # port name, slave address (in decimal)
-        instrument.mode = minimalmodbus.MODE_RTU   # rtu or ascii mode
-        datas = dict()
-        meter_id_name = dict() # mapping id to name
-
-        for meter in meters:
-            meter_id_name[meter['id']] = meter['name']
-            instrument.serial.baudrate = meter['baudrate']
-            instrument.serial.bytesize = meter['bytesize']
-            if meter['parity'] == 'none':
-                instrument.serial.parity = minimalmodbus.serial.PARITY_NONE
-            elif meter['parity'] == 'odd':
-                instrument.serial.parity = minimalmodbus.serial.PARITY_ODD
-            elif meter['parity'] == 'even':
-                instrument.serial.parity = minimalmodbus.serial.PARITY_EVEN
-            else:
-                log.error('No parity specified')
-                raise
-            instrument.serial.stopbits = meter['stopbits']
-            instrument.serial.timeout  = meter['timeout']    # seconds
-            instrument.address = meter['id']    # this is the slave address number
-
-            log.debug('Reading meter %s.' % (meter['id']))
-            start_time = time.time()
-            parameters = yaml.load(open(meter['type']), Loader=yaml.FullLoader)
-            datas[meter['id']] = dict()
-
-            for parameter in parameters:
-                # If random readout errors occour, e.g. CRC check fail, test to uncomment the following row
-                #time.sleep(0.01) # Sleep for 10 ms between each parameter read to avoid errors
-                retries = 10
-                while retries > 0:
-                    try:
-                        retries -= 1
-                        datas[meter['id']][parameter] = instrument.read_float(parameters[parameter], 4, 2)
-                        retries = 0
-                        pass
-                    except ValueError as ve:
-                        log.warning('Value Error while reading register {} from meter {}. Retries left {}.'
-                               .format(parameters[parameter], meter['id'], retries))
-                        log.error(ve)
-                        if retries == 0:
-                            raise RuntimeError
-                    except TypeError as te:
-                        log.warning('Type Error while reading register {} from meter {}. Retries left {}.'
-                               .format(parameters[parameter], meter['id'], retries))
-                        log.error(te)
-                        if retries == 0:
-                            raise RuntimeError
-                    except IOError as ie:
-                        log.warning('IO Error while reading register {} from meter {}. Retries left {}.'
-                               .format(parameters[parameter], meter['id'], retries))
-                        log.error(ie)
-                        if retries == 0:
-                            raise RuntimeError
-                    except:
-                        log.error("Unexpected error:", sys.exc_info()[0])
-                        raise
-
-            datas[meter['id']]['Read time'] =  time.time() - start_time
-
-        json_body = [
-            {
-                'measurement': 'energy',
-                'tags': {
-                    'id': meter_id,
-                    'meter': meter_id_name[meter_id],
-                },
-                'time': t_str,
-                'fields': datas[meter_id]
-            }
-            for meter_id in datas
-        ]
-        if len(json_body) > 0:
-            try:
-                self.influx_client.write_points(json_body)
-                log.info(t_str + ' Data written for %d meters.' % len(json_body))
-            except Exception as e:
-                log.error('Data not written!')
-                log.error(e)
-                raise
-        else:
-            log.warning(t_str, 'No data sent.')
-
-
-def repeat(interval_sec, max_iter, func, *args, **kwargs):
-    from itertools import count
-    import time
-    starttime = time.time()
-    for i in count():
-        if interval_sec > 0:
-            time.sleep(interval_sec - ((time.time() - starttime) % interval_sec))
-        if i % 1000 == 0:
-            log.info('Collected %d readouts' % i)
-        try:
-            func(*args, **kwargs)
-        except Exception as ex:
-            log.error(ex)
-        if max_iter and i >= max_iter:
-            return
-
-
-if __name__ == '__main__':
-
-    import argparse
-
-    parser = argparse.ArgumentParser()
-    parser.add_argument('--interval', default=60,
-                        help='Meter readout interval (seconds), default 60')
-    parser.add_argument('--meters', default='meters.yml',
-                        help='YAML file containing Meter ID, name, type etc. Default "meters.yml"')
-    parser.add_argument('--log', default='CRITICAL',
-                        help='Log levels, DEBUG, INFO, WARNING, ERROR or CRITICAL')
-    parser.add_argument('--logfile', default='',
-                        help='Specify log file, if not specified the log is streamed to console')
-    args = parser.parse_args()
-    interval = int(args.interval)
-    loglevel = args.log.upper()
-    logfile = args.logfile
-
-    # Setup logging
-    log = logging.getLogger('energy-logger')
-    log.setLevel(getattr(logging, loglevel))
-
-    if logfile:
-        loghandle = logging.FileHandler(logfile, 'w')
-        formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
-        loghandle.setFormatter(formatter)
-    else:
-        loghandle = logging.StreamHandler()
-
-    log.addHandler(loghandle)
-
-    log.info('Started app')
-
-    # Create the InfluxDB object
-    influx_config = yaml.load(open('influx_config.yml'), Loader=yaml.FullLoader)
-    client = InfluxDBClient(influx_config['host'],
-                            influx_config['port'],
-                            influx_config['user'],
-                            influx_config['password'],
-                            influx_config['dbname'])
-
-    collector = DataCollector(influx_client=client,
-                              meter_yaml=args.meters)
-
-    repeat(interval,
-           max_iter=collector.max_iterations,
-           func=lambda: collector.collect_and_store())

readings_names.yml

@@ -0,0 +1,42 @@
+voltage:
+  - Voltage__V
+  - Voltage_P1__V
+  - Voltage_P2__V
+  - Voltage_P3__V
+current:
+  - Current__A
+  - Current_P1__A
+  - Current_P2__A
+  - Current_P3__A
+  - Currents_Sum__A
+power:
+  - Power_Active__W
+  - Power_Active_P1__W
+  - Power_Active_P2__W
+  - Power_Active_P3__W
+power_apparent:
+  - Power_Apparent__VA
+  - Power_Apparent_P1__VA
+  - Power_Apparent_P2__VA
+  - Power_Apparent_P3__VA
+power_reactive:
+  - Power_Reactive__VAr
+  - Power_Reactive_P1__VAr
+  - Power_Reactive_P2__VAr
+  - Power_Reactive_P3__VAr
+powerfactor:
+  - PowerFactor
+  - PowerFactor_P1
+  - PowerFactor_P2
+  - PowerFactor_P3
+frequency:
+  - Frequency__Hz
+energy_total: Energy_Total__kWh
+energy_today: Energy_Today__kWh
+energy_yesterday: Energy_Yesterday__kWh
+energy_other:
+  - Energy_Reactive_Total__kvarh
+  - Energy_Active_Import__kWh
+  - Energy_Active_Export__kWh
+  - Energy_Reactive_Import__kvarh
+  - Energy_Reactive_Export__kvarh

sdm_setid.py

@@ -0,0 +1,100 @@
+#!/usr/bin/python3 -u
+# -*- coding: utf-8 -*-
+#
+import minimalmodbus
+import sys
+
+serialdevice = '/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK05DZIG-if00-port0'
+
+if len(sys.argv) == 2:
+    id = int(sys.argv[1])
+    idnew = int(sys.argv[2])
+    if id > 0 and id <= 247: 
+        if idnew > 0 and idnew <= 247: 
+            print("New ID out of range")
+            exit()
+        else:
+            meter_id = tmpid
+            meter_id_new = tmpidnew
+    else:
+        print("ID out of range")
+        exit()
+elif len(sys.argv) == 1:
+    id = 1
+    idnew = int(sys.argv[1])
+    if idnew <= 0 and idnew > 247:
+        print("New ID out of range")
+        exit()
+else:
+    print("Usage: sdm_setid.py [oldID] [newID]")
+    exit()
+
+exit()
+
+rs485 = minimalmodbus.Instrument(serialdevice, meter_id)
+rs485.serial.baudrate = 2400
+rs485.serial.bytesize = 8
+rs485.serial.parity = minimalmodbus.serial.PARITY_NONE
+rs485.serial.stopbits = 1
+rs485.serial.timeout = 1
+rs485.debug = False
+rs485.mode = minimalmodbus.MODE_RTU
+print(rs485)
+
+# Modbus Parity
+# Addr 0x0012
+# 4 byte float
+# 0 = 1 stop bit, no parity (default)
+# 1 = 1 stop bit, even parity
+# 2 = 1 stop bit, odd parity
+# 3 = 2 stop bits, no parity
+#rs485.write_float(0x0012, meter_id_new, number_of_registers=2)
+
+# Meter ID
+# Addr 0x0014
+# 4 byte float
+# 1-247, default 1
+rs485.write_float(0x0014, meter_id_new, number_of_registers=2)
+
+# Baud rate
+# Addr 0x00C1
+# 4 byte float
+# 0 = 2400 (default)
+# 1 = 4800
+# 2 = 9600
+# 5 = 1200
+#rs485.write_float(0x00C1, meter_id_new, number_of_registers=2)
+
+# Pulse 1 output mode
+# Addr 0x0056
+# 4 byte float
+# 1 = Import Active Energy
+# 2 = Import + Export Active Energy
+# 4 = Export Active Energy (default)
+# 5 = Import Reactive Energy
+# 8 = Export Reactive Energy
+#rs485.write_float(0x0056, meter_id_new, number_of_registers=2)
+
+# Time of scroll display
+# Addr 0xF900
+# 2 byte HEX
+# Range 0-30s
+# 0 = does not display in turns
+#rs485.write_register(0xF900, meter_id_new, number_of_registers=1)
+
+# Pulse 1 output
+# Addr 0xF910
+# 2 byte HEX
+# 0000 = 0.001 kWh/imp (default)
+# 0001 = 0.01  kWh/imp
+# 0002 = 0.1   kWh/imp
+# 0003 = 1     kWh/imp
+#rs485.write_register(0xF910, meter_id_new, number_of_registers=1)
+
+# Measurement mode
+# Addr 0xF920
+# 2 byte HEX
+# 0001 = mode 1 (total = import)
+# 0002 = mode 2 (total = import + export) (default)
+# 0003 = mode 3 (total = import - export)
+#rs485.write_register(0xF920, meter_id_new, number_of_registers=1)

setup.py

@@ -10,26 +10,25 @@ except ImportError:
     with io.open('README.md', encoding="utf-8") as f:
         readme = f.read()
 
-setup(name='energy_meter_logger',
+setup(name='ModbusLog',
       version=0.1,
-      description='Read Energy Meter data using RS485 Modbus '+
-      'and store in local database.',
+      description='Read Energy Meter data using RS485 Modbus, '+
+      'store in local InfluxDB database and publish via MQTT.',
       long_description=readme,
-      url='https://github.com/samuelphy/energy-meter-logger',
+      url='https://github.com/Flo-Kra/ModbusLogMQTT',
       download_url='',
-      author='Samuel Vestlin',
-      author_email='samuel@elphy.se',
+      author='Florian Krauter',
+      author_email='florian@krauter.at',
       platforms='Raspberry Pi',
       classifiers=[
         'Development Status :: 4 - Beta',
         'Intended Audience :: Developers',
         'License :: MIT License',
         'Operating System :: Raspbian',
-        'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3'
       ],
-      keywords='Energy Meter RS485 Modbus',
-      install_requires=[]+(['pyserial','minimalmodbus', 'influxdb', 'pyyaml'] if "linux" in sys.platform else []),
+      keywords='Energy Meter RS485 Modbus SD120 SDM630 InfluxDB',
+      install_requires=[]+(['pyserial','minimalmodbus', 'influxdb', 'pyyaml', 'paho-mqtt', ] if "linux" in sys.platform else []),
       license='MIT',
       packages=[],
       include_package_data=True,

systemctl_service_install.sh

@@ -0,0 +1,24 @@
+#!/bin/sh
+BASEDIR=$(dirname $(realpath "$0"))
+TMPFILE=modbuslog.service
+SERVICEFILE=/etc/systemd/system/modbuslog.service
+
+echo [Unit]>$TMPFILE
+echo Description=ModbusLogger>>$TMPFILE
+echo StartLimitInterval=0>>$TMPFILE
+echo >>$TMPFILE
+echo [Service]>>$TMPFILE
+echo Type=simple>>$TMPFILE
+echo Restart=always>>$TMPFILE
+echo RestartSec=1>>$TMPFILE
+echo ExecStart=$BASEDIR/modbuslog.py>>$TMPFILE
+echo User=pi>>$TMPFILE
+echo >>$TMPFILE
+echo [Install]>>$TMPFILE
+echo WantedBy=multi-user.target>>$TMPFILE
+
+sudo cp $TMPFILE $SERVICEFILE
+
+sudo systemctl daemon-reload
+sudo systemctl enable modbuslog.service
+sudo systemctl start modbuslog.service

systemctl_service_uninstall.sh

@@ -0,0 +1,7 @@
+#!/bin/sh
+SERVICEFILE=/etc/systemd/system/modbuslog.service
+
+sudo systemctl stop modbuslog.service
+sudo systemctl disable modbuslog.service
+sudo rm $SERVICEFILE
+sudo systemctl daemon-reload